onsite
Senior Data Engineer - Frontier Resourcing
Data Engineer
Senior Data Engineer responsible for designing, building and maintaining scalable data pipelines and lakehouse architectures using Python, SQL, Spark and AWS services, while collaborating in an Agile environment to support mission‑critical defence and security programmes.
About the role
Key Responsibilities
- Design, develop and maintain robust data pipelines and lakehouse solutions using Python, SQL and Apache Spark on AWS.
- Implement and manage Airflow workflows to orchestrate ETL processes and ensure data quality.
- Collaborate with data scientists, ML engineers and security teams to deliver AI, ML and geospatial intelligence solutions.
- Optimize performance, scalability and cost of data infrastructure across multi‑region deployments.
- Document architecture, data models and best practices for future maintainability.
Requirements
- 5+ years of experience in data engineering with a strong focus on cloud (AWS) and big‑data technologies.
- Hands‑on experience with Airflow or similar workflow orchestration tools.
- Solid understanding of security, compliance and data governance in a defence or high‑security environment.
- Excellent communication skills and ability to thrive in an Agile, cross‑functional team.
Skills
pythonsqlapache sparkawsairflow