onsite
Senior Engineer - Data Analytics
Software Engineer
Lead end‑to‑end data pipeline development using Airflow, Kafka, and Spark on Azure Data Platform, driving scalable analytics solutions for complex data environments.
About the role
Key Responsibilities
- Design, implement, and maintain large‑scale data pipelines with Apache Airflow, ensuring reliability and performance.
- Integrate real‑time streaming data using Apache Kafka, optimizing throughput and fault tolerance.
- Develop and optimize Spark jobs for batch and streaming analytics, leveraging Azure Databricks where appropriate.
- Collaborate with data scientists and business stakeholders to translate analytical requirements into robust data solutions.
- Monitor pipeline health, troubleshoot issues, and implement best practices for data quality and governance.
Requirements
- 5+ years of experience in data engineering with a strong focus on pipeline orchestration and streaming.
- Hands‑on experience with version control (Git) and CI/CD for data workflows.
- Excellent problem‑solving skills and ability to work in a fast‑paced, collaborative environment.