remote
Staff Research Engineer, Data Agents - Databricks
Research Engineer
Lead advanced research on enterprise AI agents, designing and scaling large language model pipelines, multi‑agent systems, and data‑centric AI solutions using Python, PyTorch, Spark, and Kubernetes.
About the role
Key Responsibilities
- Design and prototype next‑generation data agents that integrate large language models with the Databricks Lakehouse platform.
- Develop scalable training and inference pipelines using PyTorch, Apache Spark, and Kubernetes for multi‑agent systems.
- Conduct end‑to‑end research from model architecture to system optimization, publishing findings in top AI venues.
- Collaborate with product, engineering, and research teams to translate research breakthroughs into production‑ready features.
- Mentor senior engineers and guide technical direction for AI research initiatives.
Requirements
- PhD or equivalent experience in Computer Science, Machine Learning, or a related field.
- 5+ years of hands‑on experience building and deploying large language models or multi‑agent AI systems.
- Deep expertise in Python, PyTorch, and distributed computing frameworks such as Apache Spark.
- Proven track record of publishing research and delivering production‑grade AI solutions at scale.
- Strong problem‑solving skills and ability to work cross‑functionally in a fast‑moving environment.
Skills
pythonpytorchapache sparkkubernetesmachine learning