remoteonsite
Senior Databricks Data Engineer - Saarthee
Data Engineer
Senior Data Engineer specializing in Databricks and Apache Spark, designing and optimizing large‑scale data pipelines using Python, SQL, and Azure Data Factory to deliver actionable analytics for enterprise clients.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines on Databricks using Apache Spark and Python.
- Implement robust ETL/ELT processes with Azure Data Factory and orchestrate workflows with scheduling tools.
- Optimize data models, queries, and storage for performance, scalability, and cost efficiency.
- Collaborate with data scientists, analysts, and business stakeholders to translate requirements into reliable data solutions.
- Ensure data quality, governance, and security compliance across all pipelines.
Requirements
- 5+ years of professional experience in data engineering, with a focus on Databricks and Spark.
- Proficiency in Python programming and advanced SQL for data transformation and analysis.
- Hands‑on experience with Azure Data Factory or comparable cloud‑based orchestration tools.
- Strong understanding of data warehousing concepts, cloud storage (e.g., ADLS, S3), and performance tuning.
- Excellent problem‑solving skills and ability to work in fast‑paced, client‑focused environments.
Skills
databricksapache sparkpythonsql