remote
Senior Lead Platform Data Engineer
Data Engineer
Lead platform data engineering initiatives, architecting scalable Spark pipelines on Azure Synapse, and driving automated testing and CI/CD pipelines to ensure high-quality, production-ready data services.
About the role
Key Responsibilities
- Design, develop, and maintain large-scale data pipelines using Apache Spark on Azure Synapse.
- Implement automated testing frameworks to validate data quality, performance, and compliance across all stages of the data lifecycle.
- Build and manage CI/CD pipelines that streamline code integration, deployment, and monitoring for data services.
- Collaborate with cross-functional teams to translate business requirements into robust, scalable data solutions.
- Mentor junior engineers, conduct code reviews, and promote best practices in data engineering and DevOps.
Requirements
- 5+ years of experience in data engineering with a strong focus on Spark and Azure technologies.
- Proven track record of building and maintaining CI/CD pipelines for data workflows.
- Deep knowledge of automated testing strategies for data pipelines and services.
- Excellent problem‑solving skills and ability to work in a fast‑paced, collaborative environment.
Skills
apache sparkazurecicd