remote
Senior Data Engineering Lead - Optum
Software Engineer
Lead a data engineering team to design, build, and operate scalable data pipelines and platforms using Python, Spark, SQL, and AWS services, driving health‑care analytics and insights.
About the role
Key Responsibilities
- Architect, develop, and maintain end‑to‑end data pipelines that ingest, transform, and store large‑scale health data across cloud and on‑premise environments.
- Lead a team of data engineers, providing technical guidance, mentorship, and fostering best practices in code quality, testing, and documentation.
- Collaborate with data scientists, analysts, and product owners to translate business requirements into robust data models and analytics‑ready datasets.
- Implement and manage workflow orchestration using Apache Airflow, ensuring reliable scheduling, monitoring, and alerting of data jobs.
- Optimize performance and cost of data platforms on AWS (e.g., S3, Redshift, EMR, Glue) and enforce security and compliance standards.
- Drive continuous improvement by evaluating emerging technologies, establishing standards, and promoting a data‑driven culture.
Requirements
- 7+ years of hands‑on experience in data engineering, with at least 3 years in a lead or architect role.
- Strong proficiency in Python, SQL, and big‑data processing frameworks such as Apache Spark or Flink.
- Extensive experience designing and operating data pipelines on AWS (S3, Redshift, EMR, Glue, Lambda) and using Airflow for orchestration.
- Solid understanding of data modeling, ETL/ELT design patterns, and data warehousing concepts.
- Excellent communication, stakeholder management, and team‑leadership skills.
Skills
pythonsqlapache sparkawsairflow