AI Quality Architect

About the Role

As AI Quality Architect, you are the person who makes quality native to the agentic SDLC. You design the systems, standards, and intelligence layers that ensure every stage of an AI-accelerated pipeline — from requirement ingestion to autonomous deployment — is observable, trustworthy, and continuously improving. You don't retrofit testing onto AI workflows; you architect quality into them from the ground up.

How will you make an impact?

Agentic Quality Architecture

• Design the end-to-end quality architecture for agentic SDLC pipelines — spanning requirement analysis, code generation, test creation, execution, triage, and release gates
• Define how quality agents are orchestrated: which decisions they own autonomously, which require human-in-the-loop checkpoints, and how confidence thresholds govern both
• Architect multi-agent quality workflows: requirement validation agents, test generation agents, failure triage agents, and regression analysis agents working in coordinated pipelines
• Establish trust and verification models for agent-produced artifacts — test code, assertions, coverage reports, and defect analyses must all be auditable and traceable
• Own the architectural patterns for quality feedback loops between agents: how a deployment agent learns from a triage agent's findings, and how that signal improves future generation

AI-Native Test Engineering Platform

• Design and own the LLM-powered test generation platform — from natural language requirement ingestion to executable, maintainable test output
• Architect the evaluation harness that continuously measures test generation quality: coverage delta, false-positive rates, assertion accuracy, and maintenance burden over time
• Build the self-healing test infrastructure layer — agents that detect broken selectors, drifted APIs, or changed behaviors and propose or apply fixes autonomously
• Define the prompt engineering standards, context injection patterns, and RAG architectures that ground test generation agents in real codebase context
• Architect test artifact governance: versioning, ownership attribution (human vs. agent), rollback capability, and confidence scoring for every generated artifact

Quality Gates in Autonomous Pipelines

• Design intelligent, adaptive quality gates that operate at the speed of agentic CI/CD — gates that reason about risk, not just pass/fail thresholds
• Build risk-scoring models that dynamically adjust gate strictness based on change scope, code origin (human vs. AI-generated), historical failure patterns, and deployment context
• Architect the observability layer for agentic pipelines: what signals indicate a pipeline agent is making poor quality decisions, and how are those signals surfaced in real time
• Define the integration patterns between quality gates and orchestration platforms (LangChain, LlamaIndex, custom agent frameworks) used across the engineering org
• Establish rollback and circuit-breaker patterns for autonomous deployments triggered by quality signal degradation

AI Model & Agent Validation

• Build behavioral testing frameworks for validating AI agents and LLM-powered features in production — testing non-deterministic outputs with statistical rigor
• Design evaluation benchmarks for internal AI tooling: measuring agent task completion accuracy, hallucination rates, context retention, and decision quality over time
• Architect drift detection systems that identify when agent behavior changes between model versions, prompt updates, or context window shifts
• Define adversarial and edge-case testing methodologies for AI features: prompt injection resistance, boundary condition handling, and graceful degradation under distribution shift
• Partner with ML platform and data science teams to establish quality acceptance criteria for every model and agent promoted to production

Have you got what it takes?

• 12+ years in software engineering with strong depth across both development and quality engineering
• 4+ years as a hands-on principal architect or distinguished engineer with cross-org technical scope
• Demonstrated experience designing quality infrastructure used at scale — 50+ engineers, high-velocity pipelines, enterprise SLAs
• Direct production experience building or operating systems that incorporate LLMs or AI agents — not evaluations, but shipped systems
• Background in large-scale CI/CD architecture and the performance engineering domain
• Enterprise SaaS or platform engineering background; familiarity with regulated, high-uptime environments strongly preferred

Agentic AI & LLM Proficiency

• Deep, hands-on understanding of agentic AI patterns: tool use, multi-agent orchestration, planning loops, memory architectures, and human-in-the-loop design
• Expert-level prompt engineering including chain-of-thought, few-shot, RAG, and structured output techniques applied to code and test generation
• Experience designing evaluation harnesses for non-deterministic AI systems — statistical confidence, behavioral consistency, and regression detection
• Familiarity with agent frameworks (LangChain, LlamaIndex, AutoGen, CrewAI, or equivalents) and their tradeoffs in production quality pipelines

Technical Depth

• Expert in Python and TypeScript; proficient in at least one compiled language (Java, Go, C#)
• Deep knowledge of modern test frameworks across UI, API, and contract layers: Playwright, Stryker, pytest, REST Assured, Pact
• Strong CI/CD architecture skills: Jenkins, GitHub Actions, pipeline-as-code, artifact management
• Proficient with cloud-native infrastructure: Docker, Kubernetes, AWS, serverless execution environments

Leadership & Influence

• Proven ability to drive technical consensus and architectural adoption across multiple teams without direct authority
• Exceptional written communication — your ADRs, RFCs, and design docs set the standard others follow
• Effective technical mentor to senior and staff engineers; you expand the thinking of those around you
• Comfortable and credible presenting to VP and C-level stakeholders; you make complex architecture legible to executives