LLM Inference Engineer

About Us

Hippocratic AI is the leading generative AI company in healthcare. We have the only system that can have safe, autonomous, clinical conversations with patients. We have trained our own LLMs as part of our Polaris constellation, resulting in a system with over 99.9% accuracy.

About the Role

We're seeking an experienced LLM Inference Engineer to optimize our large language model (LLM) serving infrastructure. The ideal candidate has:

• Extensive hands-on experience with state-of-the-art inference optimization techniques
• A track record of deploying efficient, scalable LLM systems in production environments

What You'll Do

• Design and implement multi-node serving architectures for distributed LLM inference
• Optimize multi-LoRA serving systems
• Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality
• Implement speculative decoding and other latency optimization strategies
• Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases
• Continuously benchmark and improve system performance across various deployment scenarios and GPU types

What You Bring

Must-Have:

• Experience optimizing LLM inference systems at scale
• Proven expertise with distributed serving architectures for large language models
• Hands-on experience implementing quantization techniques for transformer models
• Strong understanding of modern inference optimization methods, including:
  • Speculative decoding techniques with draft models
  • Eagle speculative decoding approaches
• Proficiency in Python and C++
• Experience with CUDA programming and GPU optimization

Nice-to-Have:

• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
• Experience with custom CUDA kernels
• Track record of deploying inference systems in production environments
• Deep understanding of performance optimization systems

_Show us what you've built: Tell us about an LLM inference or training project that makes you proud! Whether you've optimized inference pipelines to achieve breakthrough performance, designed innovative training techniques, or built systems that scale to billions of parameters - we want to hear your story._

_Open source contributor? Even better! If you've contributed to projects like vllm, sglang, lmdeploy or similar LLM optimization frameworks, we'd love to see your PRs. Your contributions to these communities demonstrate exactly the kind of collaborative innovation we value._

Join a team where your expertise won't just be appreciated—it will be celebrated and amplified. Help us shape the future of AI deployment at scale!