onsite
Lead Data Engineer Databricks - United Utilities
Data Engineer
Lead the design and implementation of a scalable, near‑real‑time river water quality data platform using Databricks, Spark, and Python, driving data quality, performance, and automation for environmental monitoring.
About the role
Key Responsibilities
- Architect, develop and maintain a high‑throughput data pipeline on Databricks, ingesting sensor streams and batch data for real‑time water quality analysis.
- Implement Delta Lake tables, schema evolution, and data quality checks to ensure reliable, auditable data for downstream analytics.
- Collaborate with data scientists and domain experts to translate monitoring requirements into scalable Spark jobs and SQL transformations.
- Optimize Spark workloads, monitor performance, and troubleshoot production issues to meet strict latency targets.
- Mentor junior engineers, enforce coding standards, and drive best practices in data engineering and DevOps.
Requirements
- 5+ years of data engineering experience with a focus on big data platforms.
- Proven expertise in Databricks, Apache Spark, and Delta Lake.
- Strong programming skills in Python and SQL.
- Experience designing data lakes and implementing data quality frameworks.
- Excellent communication skills and a collaborative mindset.
Skills
databricksapache sparkpythonsql