remote
Senior Data Engineer - Kraken
Data Engineer
Lead end‑to‑end data pipeline development, architecting scalable solutions on AWS, leveraging Python, Spark, and Airflow to support AI/ML workloads and deliver high‑quality data products.
About the role
Key Responsibilities
- Design, build, and maintain large‑scale data pipelines that ingest, transform, and serve data for AI and analytics teams.
- Implement data models and schemas in AWS services (Redshift, S3, Glue) ensuring performance, reliability, and security.
- Automate workflow orchestration with Airflow, monitoring job health and optimizing execution times.
- Collaborate with data scientists to deploy ML models, integrating model outputs into production data streams.
- Apply best practices in data quality, lineage, and documentation to support governance and compliance.
Requirements
- 5+ years of experience in data engineering, with strong proficiency in Python and SQL.
- Hands‑on experience with Apache Spark, AWS data services, and Airflow orchestration.
- Solid understanding of data modeling, ETL design, and performance tuning.
- Experience working with ML teams and deploying model pipelines in production.
- Excellent problem‑solving skills and a collaborative mindset.
Skills
pythonsqlawsapache sparkairflow