onsite
Microsoft Databricks Engineer - Axiom Technology Group
Software Engineer
Design, build, and maintain scalable data pipelines and lakehouse solutions on Azure Databricks, collaborating with analysts to deliver reliable data products and ensure platform performance and reliability.
About the role
Key Responsibilities
- Design and implement end‑to‑end data pipelines and lakehouse architectures using Databricks on Microsoft Azure.
- Collaborate with business analysts and stakeholders to translate functional requirements into robust, production‑ready data models and datasets.
- Develop, test, and maintain Spark jobs and Python/SQL scripts for data ingestion, transformation, and enrichment.
- Monitor and optimize platform health, performance, and scalability, including cost‑effective resource provisioning.
- Implement CI/CD practices, version control, and automated testing to ensure reliable deployments.
Requirements
- 3+ years of hands‑on experience with Databricks, Azure Data Lake, and Apache Spark.
- Proficiency in Python and SQL for data processing and analytics.
- Strong understanding of cloud concepts, Azure services (e.g., ADLS, Synapse, Azure DevOps), and data security best practices.
- Experience building scalable, fault‑tolerant data pipelines and lakehouse solutions.
- Excellent problem‑solving skills and ability to work cross‑functionally with analysts and engineering teams.
Skills
databricksapache sparkpythonsql