remote
Databricks Data Engineer - E Source
Data Engineer
Senior Databricks Data Engineer building scalable, secure data pipelines and infrastructure to power machine learning and SaaS solutions using Spark, Python, SQL, and AWS services.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines on Databricks, ensuring high performance and reliability.
- Collaborate with ML engineers, data scientists, and software teams to deliver data products and support internal analytics.
- Implement data governance, security, and compliance best practices across the data platform.
- Optimize Spark workloads, tune cluster resources, and monitor pipeline health using Databricks and AWS monitoring tools.
- Automate data ingestion, transformation, and model deployment workflows with Python, SQL, and MLflow.
Requirements
- 5+ years of experience building data pipelines in a cloud environment, preferably AWS.
- Strong proficiency in Databricks, Apache Spark, Python, and SQL.
- Hands‑on experience with data lake architecture, Delta Lake, and data cataloging.
- Knowledge of data security, governance, and compliance frameworks.
- Excellent communication skills and ability to work cross‑functionally in a fast‑paced environment.
Skills
databricksapache sparkpythonsqlaws