remote
Senior Databricks Engineer
Software Engineer
Senior Databricks Engineer responsible for designing, deploying, and scaling Spark workloads on Databricks, integrating with AWS and Azure services, and orchestrating pipelines with Airflow to deliver high‑performance data solutions.
About the role
Key Responsibilities
- Architect and implement scalable Spark solutions on Databricks, ensuring optimal performance and cost efficiency.
- Develop and maintain Airflow DAGs to orchestrate data pipelines across AWS and Azure environments.
- Collaborate with data scientists and engineers to translate business requirements into robust data workflows.
- Monitor, troubleshoot, and optimize cluster autoscaling and resource utilization.
- Document best practices, coding standards, and operational procedures for the data engineering team.
Requirements
- 5+ years of experience with Databricks and Apache Spark in production.
- Strong proficiency in Python and SQL for data processing.
- Hands‑on experience with AWS services (EMR, S3, Glue) and Azure Data Lake.
- Proven track record building and managing Airflow pipelines.
- Excellent problem‑solving skills and ability to work in a fast‑paced environment.
Skills
databricksapache sparkawsazureairflow