onsite
Senior Data Engineer - Fastmarkets
Data Engineer
Lead end‑to‑end data pipeline development, architecting scalable solutions on AWS, and optimizing large‑scale ETL workflows using Python, Spark, and Airflow to support data‑driven insights for the Carbon division.
About the role
Key Responsibilities
- Design, build, and maintain robust data pipelines that ingest, transform, and load terabyte‑scale datasets into the enterprise data warehouse.
- Leverage Apache Spark and Python to develop high‑performance ETL jobs, ensuring data quality and lineage.
- Implement and manage Airflow DAGs for orchestration, monitoring, and alerting of data workflows.
- Collaborate with data scientists and analysts to translate business requirements into scalable data models and services.
- Optimize storage and compute resources on AWS (S3, Redshift, EMR) to reduce costs and improve performance.
- Document architecture, processes, and best practices for internal teams.
Requirements
- 5+ years of experience as a data engineer in a production environment.
- Strong proficiency in Python, SQL, and Spark for large‑scale data processing.
- Hands‑on experience with AWS services (S3, Redshift, EMR, Glue) and Airflow orchestration.
- Solid understanding of data warehousing concepts, dimensional modeling, and ETL best practices.
- Excellent problem‑solving skills and ability to work collaboratively in a fast‑paced, cross‑functional team.
Skills
pythonsqlapache sparkawsairflow