onsite
Senior Associate - Data Engineer - New York Life
Data Engineer
Senior Data Engineer responsible for designing, building, and optimizing data pipelines and warehouses using Python, SQL, Spark, and AWS services while ensuring data quality and supporting analytics initiatives.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and ETL processes to ingest, transform, and load large‑volume datasets.
- Implement data models and warehouse solutions on AWS (e.g., Redshift, S3, Glue) to support business intelligence and analytics.
- Collaborate with data scientists, analysts, and product teams to understand requirements and deliver reliable data services.
- Optimize performance of Spark jobs and SQL queries, ensuring low latency and high throughput.
- Monitor data quality, troubleshoot pipeline failures, and enforce best practices for data governance.
Requirements
- 5+ years of hands‑on experience in data engineering, preferably in a financial or insurance environment.
- Proficiency in Python and SQL, with strong knowledge of Apache Spark or similar distributed processing frameworks.
- Experience building and operating data solutions on AWS (Redshift, S3, Lambda, Glue, etc.).
- Solid understanding of ETL design patterns, data modeling, and relational/columnar databases.
- Ability to work in a hybrid setting, communicate effectively with cross‑functional teams, and deliver high‑quality code on schedule.
Skills
pythonsqlapache sparkaws