remote
Test Automation Lead - AI Testing - Eli Lilly
Software Engineer
We're looking for a Software Engineer focused on designing and building scalable technical solutions. This lead role requires 7+ years of relevant experience.
About the role
- Deep expertise in test automation frameworks and strategies, with the ability to extend them to validate AI and ML models, data pipelines, and intelligent applications.
- Hands-on expertise with AI evaluation methodologies, model benchmarking, and quality metrics
- A strong understanding of AI testing methodologies such as model validation, bias detection, explainability testing, and performance benchmarking.
- Ability to design end-to-end automated testing solutions for AI-enabled systems involving APIs, UI, data, and model outputs.
- Proven experience collaborating with data scientists, ML engineers, and software developers to ensure test coverage and model reliability.
- Lead with influence by promoting a culture of engineering excellence, responsible AI, and continuous improvement.
- Develop and maintain automation frameworks tailored for testing AI/ML workflows, including model training, inference, and integration.
- Create and execute automated regression, performance, and data validation tests to ensure the stability and quality of AI-driven systems.
- Implement AI-specific test approaches such as synthetic data generation, result explainability validation, and continuous model monitoring.
- Drive innovation by identifying opportunities to embed automation and intelligence into the testing lifecycle using GenAI tools and frameworks.
- Partner cross-functionally to define and track AI quality metrics (accuracy, precision, recall, fairness) and integrate them into CI/CD pipelines.
- Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architectures
- Hands-on experience in AI evaluation frameworks (RAGAS, DeepEval, LLM-as-a-Judge, LangFuse)
- Expertise in Prompt Validation, Prompt Injection Testing and Adversarial Testing (Red Teaming)
- Knowledge of Data Validation, Data Quality, Data Drift, and Training/Inference Pipeline Testing
- Deep hands-on experience with a broad range of test automation tools and technologies,
- UI and functional testing: Playwright, Cypress
- API testing: Postman
- Performance testing: k6, JMeter
- Strong understanding of supervised/unsupervised learning.
- Experience with ML platforms and tools and cloud AI services.
- Knowledge of DevOps practices and CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps.
- Excellent analytical, debugging, and communication skills with a passion for quality and innovation.
- Define, implement, and own test strategies across enterprise-level projects and programs.
- Lead and mentor global and cross-functional QA teams, promoting best practices and continuous improvement.
- Collaborate with senior leadership, product owners, architects, and development teams to ensure quality is built into all stages of the development lifecycle.