remote
Lead Data Engineer - Databricks & Azure Retail Domain - UST
Data Engineer
Lead the design and delivery of scalable, cloud‑based data platforms for a retail environment, leveraging Azure, Databricks, PySpark, and real‑time streaming to enable actionable insights across customer, product, and supply‑chain data.
About the role
Key Responsibilities
- Architect and lead the implementation of enterprise data pipelines on Azure and Databricks, ensuring high availability and performance.
- Develop batch and real‑time processing solutions using PySpark, Spark SQL, Python, and Kafka to ingest, transform, and enrich data.
- Design and maintain Lakehouse architectures with Delta Lake, optimizing storage and query performance.
- Build and optimize ETL/ELT workflows, leveraging Azure Data Factory, ADLS, and Synapse Analytics for seamless data movement.
- Collaborate with cross‑functional teams to translate business requirements into robust data solutions and provide real‑time insights.
Requirements
- Extensive experience in Azure cloud services, including Azure Data Factory, ADLS, and Synapse.
- Proficiency in Databricks, PySpark, Spark SQL, and Python for large‑scale data processing.
- Hands‑on expertise with Kafka for real‑time data streaming and Delta Lake for Lakehouse architecture.
- Strong understanding of data modeling, ETL/ELT best practices, and performance tuning.
- Excellent communication skills and ability to lead technical initiatives in a fast‑paced retail environment.
Skills
azuredatabrickspythonkafka