remote
Azure Data Technical Specialist - HCLTech
Software Engineer
Lead cloud data migration and architect enterprise‑scale Lakehouse solutions on Databricks, leveraging AWS/GCP, Unity Catalog governance, and Apache Iceberg for large‑scale analytics and AI.
About the role
Key Responsibilities
- Lead end‑to‑end migration of enterprise data to a Databricks Lakehouse architecture on AWS and GCP.
- Design, implement, and maintain data governance frameworks using Unity Catalog, ensuring lineage, access control, and compliance.
- Build and optimize large‑scale Apache Iceberg tables, managing schema evolution and performance tuning.
- Integrate Databricks with cloud storage, downstream analytics, and AI/ML pipelines.
- Collaborate with data engineers, architects, and stakeholders to define data models and best practices.
Requirements
- Proven experience with Databricks, Spark, PySpark, and Spark SQL in a production environment.
- Hands‑on expertise deploying Databricks on AWS and GCP, including Unity Catalog and Iceberg.
- Strong background in large‑scale data migration and lakehouse design.
- Solid understanding of data governance, security, and compliance standards.
- Excellent communication skills and ability to work cross‑functionally in a hybrid setting.