onsite
Senior Data Engineer - ANDHealth
Data Engineer
Senior Data Engineer building a modern data platform to support analytics, reporting, operational workflows, product integrations, and AI initiatives using Python, SQL, Airflow, AWS, and Spark.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines that ingest, transform, and load data from diverse sources into a unified data warehouse.
- Implement and optimize ETL processes using Python, SQL, and Spark, ensuring high performance and reliability.
- Leverage Airflow for workflow orchestration, monitoring, and alerting to guarantee data quality and pipeline uptime.
- Collaborate with data scientists and product teams to support AI/ML initiatives, providing clean, well‑structured data for model training and inference.
- Architect and maintain data models, schemas, and metadata management to enable self‑service analytics across the organization.
- Utilize AWS services (S3, Redshift, Glue, Lambda) to build a cost‑effective, secure, and scalable data infrastructure.
Requirements
- 5+ years of experience as a data engineer in a fast‑paced, cloud‑native environment.
- Proficiency in Python, SQL, and Spark for data processing and transformation.
- Hands‑on experience with Airflow for workflow orchestration and monitoring.
- Strong knowledge of AWS data services (S3, Redshift, Glue, Lambda) and best practices for security and cost optimization.
- Excellent problem‑solving skills, with a track record of delivering robust, production‑grade data solutions.
Skills
pythonsqlairflowaws