onsite
Data Engineer - Eden James Consulting Limited
Data Engineer
Data Engineer responsible for designing, building, and maintaining scalable data pipelines and warehouses using Python, SQL, Airflow, and cloud platforms to enable data science and BI insights for public‑sector projects.
About the role
Key Responsibilities
- Design, develop, and optimise end‑to‑end data pipelines that ingest, transform, and load large‑scale datasets from multiple sources.
- Implement and maintain data warehouse solutions on Snowflake and Azure, ensuring high performance and reliability.
- Collaborate with data scientists, analysts, and public‑sector stakeholders to translate business requirements into robust data models.
- Automate workflow orchestration using Apache Airflow, monitoring jobs and handling failures proactively.
- Apply best practices for data quality, governance, and security in compliance with public‑sector regulations.
Requirements
- Strong proficiency in Python and SQL for data manipulation and scripting.
- Hands‑on experience with cloud data platforms such as Snowflake and Azure Data Factory.
- Proven expertise in building and scheduling pipelines with Apache Airflow or similar orchestration tools.
- Solid understanding of data modelling, ETL processes, and data warehousing concepts.
- Ability to work collaboratively in cross‑functional teams and communicate technical concepts to non‑technical stakeholders.