onsite
Lead ML Inference Engineer
Lead ML Inference Engineer
As a Lead ML Inference Engineer, you will be responsible for designing and operating GPU infrastructure for model hosting, building scalable model serving systems, and implementing multi-model routing. You will own the end-to-end model lifecycle, drive inference optimization, and build self-service infrastructure platforms.
About the role
Responsibilities
- Design and operate GPU infrastructure for model hosting, including provisioning, scheduling, and cost optimization across cloud and on-premise environments
- Build and scale model serving systems using vLLM, TensorRT-LLM, Triton, or equivalent, supporting real-time inference with strong latency and availability guarantees
- Implement multi-model routing to serve multiple models across modalities (text, voice, code, vision) on shared infrastructure
- Own the model lifecycle end to end: download, deploy, serve, monitor, swap, and scale
- Drive inference optimization including quantization strategies (AWQ, GPTQ), batching, caching, and cold start reduction
- Build self-service infrastructure platforms where teams provision compute, storage, and model endpoints through APIs and control planes
- Implement infrastructure-as-code at scale using Terraform, Pulumi, or CDK
- Build observability and reliability for inference systems: SLIs/SLOs, GPU utilization monitoring, latency tracking, automated capacity planning, and alerting
- Define platform standards and governance including multi-tenant isolation, cost attribution, and resource quotas
- Lead architectural design and influence engineering direction across the AI infrastructure stack
Skills
GPUVllmTensorRT LLMTritonquantizationAWQGPTQTerraformPulumiCDKapi