remote
Data Engineer Junior or Senior - Progressive
Data Engineer
Junior or Senior Data Engineer building real‑time pipelines and ML feature services on AWS, using Python, SQL, Airflow, Spark and MLflow to support AI/ML experiments in a direct‑to‑consumer quote funnel.
About the role
Key Responsibilities
- Design, develop and maintain real‑time data pipelines that feed ML model features into production APIs.
- Own CI/CD workflows for pipeline deployment, ensuring automated testing, monitoring and rollback capabilities.
- Collaborate with data scientists to translate model requirements into scalable, cloud‑native data solutions.
- Optimize Spark jobs and SQL queries for performance and cost efficiency on AWS.
- Implement observability, logging and alerting for data pipelines and model serving endpoints.
Requirements
- 3+ years of data engineering experience with Python, SQL and Spark.
- Proficiency in Airflow for workflow orchestration and CI/CD tooling (Git, Jenkins, or equivalent).
- Hands‑on experience deploying and managing services on AWS (EC2, EMR, S3, Lambda).
- Familiarity with MLflow or similar model tracking and serving frameworks.
- Strong problem‑solving skills and ability to work in a fast‑paced, cross‑functional team.
Skills
pythonsqlairflowawsmlflowcicd