remote
Associate Data Engineering Manager - Optum
Engineering Manager
Lead the design and delivery of batch and real‑time data pipelines using Python, PySpark, and Databricks on Azure, while managing a small team of data engineers and collaborating with cross‑functional stakeholders.
About the role
Key Responsibilities
- Architect, develop, and maintain scalable batch and streaming data pipelines using PySpark, Databricks, and Kafka on Azure.
- Lead a team of 2–4 data engineers, providing mentorship, code reviews, and performance coaching.
- Collaborate with data scientists, product owners, and business stakeholders to translate requirements into robust data solutions.
- Ensure data quality, reliability, and security through automated testing, monitoring, and documentation.
- Drive continuous improvement of data engineering processes, tooling, and best practices.
Requirements
- 3+ years of experience in data engineering with a strong focus on Python and PySpark.
- Hands‑on experience with Databricks, Kafka, and Azure Data Lake / Synapse.
- Proven ability to lead and mentor a small engineering team.
- Strong analytical skills and a passion for delivering high‑quality data solutions.
- Excellent communication skills and ability to work cross‑functionally.
Skills
pythondatabrickskafkaazure