remote
Lead Software Engineer - Data Engineer - CGI
Data Engineer
Lead a data engineering team to design, build, and optimize large‑scale data pipelines and warehouses using Python, SQL, Spark, and Azure services.
About the role
Key Responsibilities
- Architect and develop end‑to‑end data pipelines for ingestion, transformation, and storage across on‑premise and cloud environments.
- Lead a team of engineers, providing technical guidance, code reviews, and mentorship.
- Design scalable data models and data warehouse solutions to support analytics and reporting.
- Implement real‑time streaming solutions using Kafka and Spark Structured Streaming.
- Collaborate with data scientists, business analysts, and stakeholders to translate requirements into robust data solutions.
- Ensure data quality, security, and compliance through automated testing and monitoring.
Requirements
- 5+ years of hands‑on experience in data engineering, with a strong focus on Python and SQL.
- Proficiency in Apache Spark (batch and streaming) and cloud data services, preferably Azure Data Factory, Azure Synapse, or similar.
- Experience designing and maintaining data warehouses or data lakes, including ETL/ELT processes.
- Solid understanding of messaging systems such as Kafka and event‑driven architectures.
- Demonstrated leadership skills, including team mentorship and project ownership.
Skills
pythonsqlapache sparkkafka