onsite
Senior Databricks Consultant - Deloitte GmbH
Software Engineer
Lead complex data engineering and analytics projects on Databricks, designing lakehouse architectures, optimizing Spark workloads, and delivering advanced ML solutions using Python, Scala, and Azure services.
About the role
Key Responsibilities
- Architect and implement end‑to‑end lakehouse solutions on Databricks, leveraging Delta Lake and Azure Data Lake Storage.
- Develop and tune Spark jobs in Python and Scala to process terabyte‑scale data streams and batch pipelines.
- Integrate MLflow for experiment tracking, model registry, and deployment pipelines.
- Collaborate with data scientists to translate ML models into production‑ready Spark UDFs and notebooks.
- Provide technical mentorship to junior engineers and conduct code reviews.
- Optimize cluster usage, cost, and performance through resource tuning and job scheduling.
Requirements
- 5+ years of experience in big data engineering with a focus on Databricks and Spark.
- Hands‑on experience with Delta Lake, MLflow, and Azure Data Lake services.
- Strong understanding of data lakehouse architecture and best practices.
- Excellent problem‑solving skills and ability to communicate complex concepts to stakeholders.
Skills
databricksapache sparkpythonscalasqlmlflow