onsite
Senior Data Engineer - Frontier Resourcing Ltd
Data Engineer
Senior Data Engineer responsible for designing, building, and maintaining scalable data pipelines and lakehouse architectures in a high‑security environment, leveraging Python, SQL, AWS, Airflow, and Spark to support AI and machine learning initiatives.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines and lakehouse solutions using Python, SQL, and Spark on AWS.
- Implement and manage Airflow workflows to orchestrate ETL processes for mission‑critical data assets.
- Collaborate with data scientists and ML engineers to provide clean, high‑quality data for AI and machine learning projects.
- Ensure data security, compliance, and integrity across all stages of the data lifecycle.
- Monitor, troubleshoot, and optimize pipeline performance, implementing best practices for scalability and reliability.
Requirements
- 5+ years of experience in data engineering within a regulated or high‑security domain.
- Hands‑on experience with AWS services (S3, Redshift, Glue, EMR) and Airflow orchestration.
- Strong understanding of data lake architecture, data modeling, and metadata management.
- Excellent problem‑solving skills and ability to work collaboratively in a multidisciplinary team.
Skills
pythonsqlawsmachine learningapache spark