remote
Data Engineer - Duxxel
Data Engineer
Design and optimize scalable data pipelines on Azure Databricks using PySpark, Delta Lake, SQL, and Python to process high‑volume enterprise data with performance, reliability, and quality assurance.
About the role
Key Responsibilities
- Design, build, and optimize scalable data pipelines using Azure Databricks, PySpark, and Delta Lake for high‑volume data processing and transformation.
- Develop and maintain complex SQL and Python solutions for data transformation, validation, and quality assurance across large enterprise datasets.
- Implement robust data engineering workflows on Azure, ensuring performance, scalability, and reliability.
- Configure and manage Azure Databricks clusters, notebooks, and workflows to support continuous integration and deployment of data solutions.
- Collaborate with data scientists and analysts to translate business requirements into efficient data pipelines and reporting solutions.
Requirements
- Strong hands‑on experience with Azure Databricks, including notebooks, workflows, and cluster configuration.
- Solid expertise in Delta Lake and Azure data services such as Azure Data Lake Storage.
- Proficiency in PySpark, SQL, and Python for data transformation and automation.
- Experience with performance tuning, data quality, and monitoring of data pipelines.
- Excellent problem‑solving skills and ability to work in a fast‑paced, collaborative environment.