onsite
Senior Data Engineer - Abacus Insights
Data Engineer
Lead the design and implementation of scalable data pipelines in a health‑care analytics environment, leveraging Python, Spark, and AWS to transform and model data for actionable insights.
About the role
Key Responsibilities
- Architect, develop, and maintain large‑scale data pipelines using Python, Spark, and AWS services (S3, Redshift, Glue).
- Design and enforce robust data models and schemas to support analytics and reporting across health‑plan stakeholders.
- Implement and monitor ETL workflows with Airflow, ensuring data quality, lineage, and compliance with regulatory standards.
- Collaborate with data scientists, product managers, and business analysts to translate requirements into efficient, production‑ready solutions.
- Optimize performance and cost of data processing jobs, applying best practices in partitioning, caching, and resource allocation.
Requirements
- 5+ years of experience in data engineering, preferably in the health‑care or insurance sector.
- Proficiency in Python, SQL, and Apache Spark for large‑scale data processing.
- Hands‑on experience with AWS data services (S3, Redshift, Glue, Athena).
- Strong knowledge of data modeling, ETL design, and workflow orchestration (Airflow).
- Excellent problem‑solving skills and a passion for building reliable, scalable data infrastructure.
Skills
pythonsqlapache sparkawsairflow