onsite
Data Engineer - Everything Managed Group
Data Engineer
Data Engineer responsible for designing, building, and maintaining scalable data pipelines and infrastructure on AWS, leveraging Python, SQL, Spark, and Airflow to transform raw data into actionable insights for cross‑functional teams.
About the role
Key Responsibilities
- Design, develop, and maintain robust ETL pipelines using Python, SQL, and Spark to ingest, transform, and load data into the EMG data platform.
- Implement and manage data workflows with Airflow, ensuring reliability, scalability, and timely execution.
- Collaborate with data scientists, analysts, and product teams to understand requirements and deliver analytical solutions.
- Optimize data storage and query performance on AWS services such as Redshift, S3, and Glue.
- Monitor pipeline health, troubleshoot issues, and continuously improve data quality and processing efficiency.
Requirements
- 3+ years of experience as a Data Engineer or similar role.
Skills
pythonsqlawsapache sparkairflow