onsite
Senior Data Engineer PySpark, Azure Data Factory - Epathusa
Data Engineer
Senior Data Engineer responsible for designing and operating scalable data pipelines on Azure, leveraging PySpark, Azure Data Factory, and related services to deliver BI solutions that improve energy forecasting and operational efficiency.
About the role
Key Responsibilities
- Collaborate with business stakeholders to capture requirements and translate them into end‑to‑end data and BI solutions.
- Design, develop, and maintain high‑performance data pipelines using PySpark and Azure Data Factory.
- Integrate data from Azure Data Lake, Synapse Analytics, and Databricks to support reporting, analytics, and forecasting workloads.
- Implement data quality, monitoring, and error‑handling mechanisms to ensure reliable pipeline execution.
- Optimize data models and queries for performance and cost efficiency in the Azure ecosystem.
Requirements
- 10+ years of professional experience in data engineering or related fields.
- Strong expertise in PySpark programming and Azure Data Factory orchestration.
- Hands‑on experience with Azure Data Lake, Synapse Analytics, and Databricks.
- Proficiency in SQL and data modeling concepts.
- Ability to work cross‑functionally, communicate technical concepts clearly, and solve complex problems in the energy sector.