remote
Lead Data Engineer - Databricks & AI - Nexzentek
Data Engineer
Lead the design, deployment, and optimization of enterprise‑scale Databricks pipelines, driving AI solutions for a transportation client while mentoring a high‑performing data engineering team.
About the role
Key Responsibilities
- Architect and implement scalable Databricks pipelines for large‑volume data ingestion, transformation, and analytics.
- Lead a team of data engineers, providing mentorship, code reviews, and best‑practice guidance.
- Integrate machine learning workflows using MLflow, ensuring reproducibility and model governance.
- Collaborate with data scientists and product owners to translate business requirements into robust data solutions.
- Optimize Spark jobs for performance and cost, leveraging Delta Lake and advanced partitioning strategies.
Requirements
- 10–15 years of experience in data engineering with a strong focus on Databricks and Spark.
- Proficiency in Python, SQL, and Spark SQL, with hands‑on experience building ETL pipelines.
- Deep knowledge of Delta Lake, MLflow, and cloud data platform best practices.
- Demonstrated leadership in managing and scaling engineering teams.
- Excellent communication skills and ability to translate technical concepts to non‑technical stakeholders.
Skills
databrickspythonapache sparksqlmlflow