onsite
Azure Data Factory Lead
Software Engineer
Lead the design, development, and delivery of end‑to‑end data pipelines on Azure Data Factory, leveraging PySpark, Snowflake, and GCP while driving AI/ML and Generative AI initiatives.
About the role
Key Responsibilities
- Architect and lead the implementation of scalable data pipelines using Azure Data Factory, PySpark, and Snowflake.
- Design and optimize data models and ETL processes for big‑data platforms such as Hadoop and GCP.
- Collaborate with data scientists and AI engineers to integrate AI/ML and Generative AI models into production workflows.
- Mentor and guide a team of data engineers, ensuring best practices in code quality, testing, and documentation.
- Monitor performance, troubleshoot issues, and continuously improve pipeline reliability and cost efficiency.
Requirements
- 5+ years of hands‑on experience with Azure Data Factory and PySpark development.
- Strong expertise in Snowflake, Hadoop ecosystems, and Google Cloud Platform services.
- Proficiency in Python for data engineering and AI/ML integration.
- Demonstrated experience delivering AI/ML or Generative AI solutions in production.
- Excellent leadership, communication, and problem‑solving skills.
Skills
snowflakepythonhadoopgenerative ai