onsite
Data Engineer II - Travelers
Data Engineer
Data Engineer II responsible for designing, building, and maintaining scalable data pipelines using Python, Spark, and AWS services to support analytics and reporting across the organization.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines using Python and Apache Spark to ingest, transform, and load data from diverse sources into data lakes and warehouses.
- Implement and optimize ETL processes on AWS (Glue, Redshift, S3) ensuring high performance, reliability, and data quality.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
- Develop and enforce data modeling standards, including dimensional modeling and schema design for analytical workloads.
- Monitor pipeline health, troubleshoot issues, and implement automated alerts and recovery mechanisms.
Requirements
- 3+ years of experience in data engineering with a strong focus on Python and Spark.
- Proficiency in SQL and experience with relational and columnar databases (Redshift, Snowflake).
- Hands‑on experience with AWS data services (Glue, S3, Redshift, Athena).
- Solid understanding of ETL best practices, data quality, and performance tuning.
- Excellent problem‑solving skills and ability to work collaboratively in a fast‑paced environment.
Skills
pythonsqlapache sparkaws