remote
Databricks Data Engineer - Horizon Industries Ltd
Data Engineer
Seasoned Data Engineer to architect and maintain scalable data pipelines on Databricks, leveraging Spark, Python, SQL and Delta Lake to deliver high‑quality, governed data assets for analytics and machine learning.
About the role
Key Responsibilities
- Design, develop and optimize data pipelines on Databricks using Spark, Python and SQL.
- Implement Delta Lake best practices for ACID transactions, schema evolution and data versioning.
- Collaborate with data scientists and business analysts to understand data requirements and deliver clean, governed datasets.
- Monitor pipeline performance, troubleshoot issues and implement automated alerts.
- Document architecture, code, and data lineage for compliance and knowledge transfer.
Requirements
- 3+ years of experience building data pipelines in a cloud environment.
- Strong understanding of data lake concepts, ETL patterns and data governance.
- Experience with cloud data services (Azure Data Lake, AWS S3, or GCP Storage) is a plus.
- Excellent problem‑solving skills and ability to work independently in a short‑term project.
Skills
databricksapache sparkpythonsql