Senior Databricks Engineer - Chubb Fire & Security

Key Responsibilities

• Design and implement robust lakehouse architectures on Databricks, leveraging Delta Lake and Unity Catalog for data storage and governance.
• Develop, test, and maintain Delta Live Tables pipelines to deliver reliable, declarative data transformation workflows.
• Apply performance‑tuning techniques such as Z‑ordering, liquid clustering, auto‑optimize, and strategic partitioning to maximize Spark job efficiency.
• Orchestrate end‑to‑end data workflows using Databricks Workflows, ensuring reliable scheduling, monitoring, and error handling.
• Collaborate with data scientists, analysts, and security teams to enforce fine‑grained access controls and maintain data lineage across the platform.

Requirements

• 5+ years of hands‑on experience with Databricks, Apache Spark, and Delta Lake in a production environment.
• Proven expertise in building and managing Delta Live Tables pipelines and using Unity Catalog for data governance.
• Strong knowledge of performance optimization techniques for Spark and Delta tables, including Z‑ordering and clustering.
• Experience with cloud platforms (AWS preferred) and infrastructure‑as‑code tools for provisioning Databricks workspaces.
• Excellent problem‑solving skills and ability to work cross‑functionally in an agile development setting.