Machine Learning Engineer, LLM Infrastructure

About the Role

Scale is seeking a Machine Learning Engineer to join the Machine Learning Infrastructure team. In this role, you will be instrumental in building systems that accelerate the development and deployment of machine learning models, with a particular focus on large language models (LLMs). You will collaborate closely with Machine Learning researchers and internal users to understand their requirements and apply your domain expertise to create high-performance and reusable APIs.

The ideal candidate possesses strong ML fundamentals and the ability to apply them in real production settings. This position specifically emphasizes optimizing inference and fine-tuning processes for LLMs. Comfort with infrastructure, large-scale system design, and the diagnosis of both model performance and system failures is essential.

Responsibilities

• Build highly available, observable, performant, and cost-effective APIs for model inference and fine-tuning for LLMs.
• Engage with ML researchers and stay updated on the latest trends from industry and academia.
• Participate in the team’s on-call process to ensure the availability of services.
• Own projects end-to-end, from requirements gathering, scoping, and design to implementation, within a highly collaborative and cross-functional environment.
• Exercise good judgment in building systems and tools, and know when to make build vs. buy tradeoffs, with an eye for cost efficiency.

Requirements

Ideally you'd have:

• 2+ years of experience building machine learning training pipelines or inference services in a production setting.
• Experience with LLM deployment, fine-tuning, training, prompt engineering, etc.
• Experience with LLM inference latency optimization techniques, such as kernel fusion, quantization, dynamic batching, etc.
• Experience with CUDA, model compilers, and other model-specific optimizations.

Nice to haves:

• Experience working with a cloud technology stack (e.g., AWS or GCP).
• Experience building, deploying, and monitoring complex microservice architectures.
• Experience with Python, Docker, Kubernetes, and Infrastructure as Code (e.g., Terraform).