onsite
Staff Data Engineer - Data Idols
Data Engineer
Lead the design and scaling of production data pipelines using Python, Spark, and Airflow on AWS, driving engineering excellence across a retail data platform.
About the role
Key Responsibilities
- Own end‑to‑end production data pipelines, ensuring reliability, performance, and scalability for high‑volume retail data.
- Architect and maintain a modern data lake on AWS, applying best practices in data modeling, partitioning, and governance.
- Lead code reviews, enforce coding standards, and mentor junior engineers to elevate overall engineering quality.
- Collaborate with data scientists, product owners, and stakeholders to translate business requirements into robust data solutions.
- Implement CI/CD pipelines, automated testing, and monitoring for data workflows using Airflow and related tooling.
Requirements
- 5+ years of experience building large‑scale data pipelines in a production environment.
- Strong proficiency in Python, SQL, and Apache Spark for data processing.
- Hands‑on experience with Airflow, AWS services (S3, Redshift, Glue, EMR), and data lake architecture.
- Solid understanding of data modeling, ETL best practices, and data governance.
- Excellent communication skills and a proven ability to mentor and influence cross‑functional teams.
Skills
pythonsqlapache sparkairflowaws