remote
Test Architect / Tech Lead - CTI MD Tech GRA/GSC - Eli Lilly
Software Engineer
We're looking for a Software Engineer focused on designing and building scalable technical solutions. This lead role requires 7+ years of relevant experience.
About the role
- Reusable Test Framework Development — design, build, and own reusable test automation frameworks and shared components (libraries, harnesses, fixtures, evaluation pipelines) that can be adopted consistently across GRA and GSC rather than rebuilt per program.
- Application Architecture & Design — design and build the application architecture for GRA/GSC platforms as a tech lead — defining technical approach, component design, and integration patterns alongside the test and quality strategy.
- Delivery Ownership — act as tech lead for key technical initiatives, owning end-to-end delivery — from architecture and build through testing and release — and holding accountability for outcomes, not just quality gates.
- Automation Enablement — drive test automation efforts across GRA and GSC, identifying opportunities to replace manual and duplicated testing effort with shared, scalable automation.
- AI & Agentic Quality Engineering — build and own evaluation frameworks for generative AI and agentic systems — groundedness validation, hallucination detection, relevance/accuracy testing, and RAG and LLM evaluation — using tools such as RAGAS, DeepEval, and LLM-as-a-Judge approaches.
- Test Initiative Leadership — define and drive test initiatives end to end — scoping, prioritizing, and delivering automation and quality-engineering improvements across concurrent teams and workstreams.
- Stakeholder Management — partner with engineering leads, product owners, and quality/compliance stakeholders to align on architecture, delivery plans, test strategy, and adoption of shared frameworks.
- Observability & Trace Analysis — establish agent observability and trace-analysis practices to diagnose failures in multi-agent and agentic workflows, and feed findings back into architecture, framework, and test design.
- Synthetic Test Data & Coverage Strategy — build synthetic test data generation capabilities and apply risk-based prioritization to determine test depth and coverage across critical workflows.
- Talent Development & Mentorship — mentor test engineers and technical leads, building a team capable of operating independently on architecture, framework design, automation, and AI evaluation.
- Proven track record architecting and scaling test automation frameworks across multiple concurrent programs.
- Demonstrated experience designing application architecture and leading end-to-end delivery of key technical initiatives as a tech lead.
- Demonstrated experience driving technical direction for AI-driven or agentic platforms, not solely test execution.
- Experience setting quality/validation standards enforced consistently across distributed or vendor-delivered teams.
- Track record of operating with seniority across functional boundaries in ambiguous, fast-scaling transformation environments.