remoteonsite
Senior Data Engineer - Newleaf HR Solutions
Data Engineer
Senior Data Engineer with 5+ years experience designing and operating scalable data pipelines on Azure, leveraging Data Factory, Databricks, PySpark, and Delta Lake within a medallion architecture.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines using Azure Data Factory and Azure Databricks.
- Implement robust ETL/ELT processes with Apache PySpark, Python, and SQL to ingest, transform, and load data from diverse sources.
- Apply Delta Lake and medallion architecture (Bronze, Silver, Gold) to ensure data quality, lineage, and performance.
- Optimize pipeline performance, monitor jobs, and troubleshoot issues in a hybrid work environment.
- Collaborate with data analysts, scientists, and stakeholders to translate business requirements into technical solutions.
Requirements
- 5+ years of professional experience in data engineering, preferably on Azure platforms.
- Strong hands‑on expertise with Azure Data Factory, Azure Databricks, and Apache PySpark.
- Proficiency in Python and SQL for data manipulation and scripting.
- Experience building ETL/ELT pipelines and implementing Delta Lake with medallion architecture.
- Ability to work independently, manage short‑term contracts, and deliver high‑quality solutions on schedule.