onsite
Azure Data Lead - HCLTech
Software Engineer
Lead a team of data engineers to design and scale a metadata‑driven ingestion framework on Azure, leveraging Databricks, Spark, and streaming technologies to build reusable, high‑performance Lakehouse pipelines.
About the role
Key Responsibilities
- Design and implement a metadata‑driven data ingestion framework (batch, streaming, CDC) for Azure Lakehouse platforms.
- Define and enforce architectural standards, best practices, and coding guidelines for data engineering teams.
- Lead a team of data engineers, providing mentorship, code reviews, and technical direction.
- Develop scalable pipelines using Databricks, Apache Spark, Python, and Kafka to ingest and transform data from diverse enterprise sources.
- Implement CI/CD pipelines, automated testing, and monitoring to ensure reliability and maintainability.
Requirements
- 5+ years of hands‑on experience in data engineering, with deep expertise in Azure, Databricks, and Spark.
- Proficiency in Python or Scala, SQL, and streaming technologies such as Kafka.
- Strong background in designing and operating large‑scale, metadata‑driven ingestion frameworks.
- Experience leading technical teams, driving best practices, and delivering production‑grade data solutions.
- Solid understanding of CI/CD, version control, and cloud‑native monitoring tools.
Skills
azuredatabricksapache sparkpythonsqlkafkacicd