remote
Azure Databricks Engineer - Capgemini
Software Engineer
Senior engineer driving big‑data solutions on Azure Databricks, building scalable Spark pipelines, integrating data lakes, and delivering analytics and ML workflows with Python, Scala, and SQL.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines on Azure Databricks using Spark, Python, and Scala.
- Implement Delta Lake tables and manage data lake storage in Azure Data Lake Storage Gen2.
- Collaborate with data scientists to deploy ML models using MLflow and integrate them into production workflows.
- Optimize performance and cost of Databricks clusters, applying best practices for scaling and resource management.
- Document architecture, code, and operational procedures; mentor junior team members.
Requirements
- 4+ years of hands‑on experience with Azure Databricks and Spark.
- Strong proficiency in Python and Scala for data engineering tasks.
- Experience with Delta Lake, Azure Data Lake Storage, and SQL-based data transformations.
- Solid understanding of cloud security, governance, and cost‑optimization in Azure.
- Excellent communication skills and a customer‑focused mindset.