onsite
Data Engineer - Airflow
Data Engineer
Lead data pipeline development using Airflow on AWS, designing scalable ETL workflows, optimizing data flows, and ensuring data quality for analytics and machine learning initiatives.
About the role
Key Responsibilities
- Design, build, and maintain robust Airflow DAGs to orchestrate data pipelines across AWS services.
- Develop and optimize SQL queries and data transformations for large datasets.
- Collaborate with data scientists and analysts to understand data requirements and deliver reliable data assets.
- Implement monitoring, alerting, and logging for pipeline health and performance.
- Ensure data security, compliance, and best practices in data handling.
Requirements
- Proven experience with Python and Airflow in a production environment.
- Strong knowledge of AWS services (S3, Redshift, Glue, Lambda, EMR).
- Solid SQL skills and familiarity with data modeling.
- Experience with CI/CD pipelines and version control (Git).
- Excellent problem‑solving skills and ability to work in an agile team.
Skills
pythonairflowawssql