remote
R4 Data Engineering - Eli Lilly
Software Engineer
Lead the design and architecture of Databricks‑based AI and Agentic systems, driving scalable data engineering solutions with Python and Spark to accelerate life‑changing research and development.
About the role
Key Responsibilities
- Architect end‑to‑end Databricks pipelines for large‑scale data ingestion, processing, and model deployment.
- Design and implement AI and Agentic system frameworks that integrate seamlessly with existing data platforms.
- Collaborate with data scientists, ML engineers, and product teams to translate business requirements into robust, production‑ready solutions.
- Lead performance tuning, cost optimization, and security best practices across the data stack.
- Mentor and guide junior engineers, fostering a culture of continuous learning and innovation.
Requirements
- 5+ years of experience in data engineering with a strong focus on Databricks and Spark.
- Proficiency in Python, SQL, and distributed data processing.
- Deep understanding of AI/ML workflows, including model training, validation, and deployment.
- Experience designing scalable, secure, and maintainable data architectures.
- Excellent communication skills and a collaborative mindset.