onsite
Senior Data Engineer SDE-2 - Darwinbox
Data Engineer
Senior Data Engineer responsible for building scalable data pipelines, optimizing performance, and delivering analytics solutions using Python, Spark, and AWS services to drive data‑driven innovation.
About the role
Key Responsibilities
- Design, develop, and maintain large‑scale data pipelines using Python and Apache Spark on AWS.
- Implement robust ETL processes, ensuring data quality, consistency, and reliability across multiple data sources.
- Collaborate with data scientists and product teams to translate business requirements into efficient data models and analytics solutions.
- Optimize query performance and storage strategies in data warehouses (e.g., Redshift, Snowflake).
- Monitor pipeline health, troubleshoot issues, and continuously improve system scalability and resilience.
Requirements
- 5+ years of experience in data engineering or related roles.
- Strong proficiency in Python, SQL, and Spark for data processing.
- Hands‑on experience with AWS services such as S3, EMR, Glue, and Redshift.
- Solid understanding of data modeling, ETL best practices, and data warehousing concepts.
- Excellent problem‑solving skills and ability to work collaboratively in a fast‑paced environment.
Skills
pythonsqlapache sparkaws