remoteonsite
Azure Databricks Engineer ADB AI/ ML - Persistent Systems
Software Engineer
Lead data engineering initiatives on Azure Databricks, building scalable Spark pipelines and AI/ML models to deliver actionable insights for enterprise clients.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines on Azure Databricks using Spark and Python.
- Implement data ingestion from diverse sources into Azure Data Lake and orchestrate workflows with Azure Data Factory.
- Collaborate with data scientists to deploy ML models, ensuring reproducibility and performance at scale.
- Optimize Spark jobs for cost, latency, and resource utilization across large datasets.
- Document architecture, code, and best practices; mentor junior engineers on Databricks and Spark.
Requirements
- 3+ years of experience with Azure Databricks, Spark, and Python.
- Strong background in data engineering, ETL, and data lake architecture.
- Hands‑on experience building and deploying ML models in production.
- Proficiency with SQL, Azure Data Factory, and Azure Data Lake Storage.
- Excellent problem‑solving skills and ability to work in a fast‑paced, client‑facing environment.
Skills
apache sparkpythonmachine learning