remote
Lead I - Data Engineering - UST
Software Engineer
Lead Data Engineer driving scalable Azure-based data pipelines, leveraging Azure Databricks, Azure Data Factory, and Apache Spark to deliver high‑quality data integration and transformation solutions for business and data science teams.
About the role
Key Responsibilities
- Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory to ingest data from databases, APIs, flat files, and cloud storage.
- Build and maintain scalable data processing solutions with Azure Databricks and Apache Spark, ensuring performance and reliability.
- Collaborate closely with business stakeholders, data scientists, and engineering teams to translate requirements into robust data integration workflows.
- Implement data quality checks, monitoring, and alerting to guarantee pipeline integrity and timely delivery.
- Document architecture, processes, and best practices for future maintenance and knowledge transfer.
Requirements
- Proven experience as an Azure Data Engineer with deep knowledge of Azure Data Factory and Azure Databricks.
- Strong proficiency in Apache Spark, SQL, and data modeling.
- Hands‑on experience designing and optimizing ETL/ELT pipelines for large datasets.
- Excellent communication skills and ability to work cross‑functionally with data scientists and business analysts.
- Experience with version control, CI/CD for data pipelines, and cloud security best practices.