remote
Databricks Architect - Property & Casualty - Derex Technologies inc
Software Engineer
Lead the design and implementation of scalable data pipelines on Databricks for P&C clients, leveraging Spark, Delta Lake, and cloud services to deliver high‑performance analytics solutions.
About the role
Key Responsibilities
- Architect and develop end‑to‑end data pipelines on Databricks, ensuring performance, reliability, and scalability for P&C data workloads.
- Collaborate with data scientists, analysts, and business stakeholders to translate business requirements into robust Spark/Scala/Python solutions.
- Implement Delta Lake best practices for ACID transactions, schema evolution, and data versioning.
- Optimize Spark jobs, tune cluster configurations, and monitor job performance using Azure/AWS monitoring tools.
- Provide technical guidance and mentorship to junior engineers and data teams.
Requirements
- 5+ years of experience architecting data solutions on Databricks with strong proficiency in Spark, Scala, and Python.
- Deep knowledge of Delta Lake, data lakehouse architecture, and data governance principles.
- Hands‑on experience with cloud platforms (Azure Databricks or AWS Glue/EMR) and related services.
- Solid SQL skills and experience with data modeling for analytical workloads.
- Excellent communication skills and a proven ability to work in cross‑functional teams.
Skills
databricksapache sparkpythonscala