onsite
Databricks Platform Engineer - Union Investment
Devops Engineer
Design, implement, and maintain scalable Databricks data platforms on Azure, leveraging Spark, Python, and SQL while ensuring robust CI/CD pipelines and infrastructure-as-code practices.
About the role
Key Responsibilities
- Architect, build, and operate secure, high‑performance Databricks clusters on Azure for data engineering and analytics workloads.
- Develop and maintain ETL pipelines using Apache Spark, Python, and SQL, ensuring data quality and reliability.
- Implement CI/CD workflows and infrastructure‑as‑code (Terraform) to automate provisioning, testing, and deployment of the data platform.
- Collaborate with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
- Monitor platform performance, troubleshoot issues, and continuously optimize cost and resource utilization.
Requirements
- Strong experience with Databricks and Apache Spark in production environments.
- Proficiency in Python and SQL for data processing and pipeline development.
- Hands‑on experience with Azure services (e.g., Azure Data Lake, Azure Synapse, Azure DevOps).
- Solid understanding of CI/CD principles and infrastructure‑as‑code tools such as Terraform or ARM templates.
- Excellent problem‑solving skills and ability to work autonomously in a fast‑paced, collaborative team.
Skills
databricksapache sparkpythonsqlazurecicdterraform