remote
Staff Site Reliability Engineer AI Platform - Manychat
Site Reliability Engineer
You’ll need: It would be great if you have:
About the role
WHAT YOU'LL DO 🚀
- Own reliability and performance of our AI infrastructure: AI Gateway, inference services, and integrations with Amazon Bedrock, Azure OpenAI, and other LLM providers.
- Design and evolve the AI Gateway: routing, failover between providers, rate limiting, caching, and guardrails.
- Build observability for AI systems: latency/throughput/error SLOs per model and provider, token-level metrics, quality and drift signals.
- Drive cost optimization and FinOps for AI workloads: per-feature cost visibility, model right-sizing, caching strategies, provider mix.
- Run capacity planning and incident response for inference services; write and improve runbooks and postmortems.
- Scale AI expertise across the org: set standards, review designs, and coach teams shipping LLM-backed features.
TO SHINE IN THIS ROLE 💥
You’ll need:
- 5+ years in SRE / platform / infrastructure engineering, including production ownership at significant scale.
- Hands-on experience operating LLM-backed systems in production: provider APIs (Bedrock, OpenAI, Anthropic, or similar), inference pipelines, self-hosted or managed model serving.
- Deep cloud-native background: AWS, Kubernetes, Terraform/IaC, CI/CD.
- Strong observability practice (Prometheus/Grafana, OpenTelemetry, or equivalent) and experience defining SLOs for non-deterministic systems.
- Proven cost-optimization work: you can show where you cut cloud or inference spend and how you made cost visible.
- Staff-level influence: you've set technical direction beyond your own team and brought others along
It would be great if you have:
- Experience building or operating an LLM gateway/proxy (e.g., LiteLLM, Kong AI Gateway, custom).
- Experience with GPU workload optimization, quantization, or serving frameworks (vLLM, TGI, Triton).
- Experience with eval pipelines and quality monitoring for LLM outputs.
Why this role
- Green-field ownership: the AI Platform is young; you'll shape its architecture, standards, and roadmap.
- Real scale and real stakes: AI features sit in the critical path of customer-facing automation.
- Direct partnership with the Head of Infrastructure; high autonomy and visibility.