onsite
Data Tech Lead
Software Engineer
Lead data engineering initiatives, architecting scalable pipelines on AWS using Apache Kafka and Spark, while driving CI/CD practices and Agile delivery.
About the role
Key Responsibilities
- Design, develop, and maintain large‑scale data pipelines on AWS, leveraging Apache Kafka for real‑time ingestion and Apache Spark for batch and streaming processing.
- Implement and manage CI/CD workflows to automate build, test, and deployment of data services, ensuring rapid and reliable delivery.
- Collaborate with cross‑functional teams in an Agile environment, translating business requirements into technical solutions and prioritizing backlog items.
- Mentor and guide junior engineers, fostering best practices in coding, testing, and documentation.
- Monitor system performance, troubleshoot issues, and optimize resource utilization across the data platform.
Requirements
- Proven experience leading data engineering teams and delivering production‑grade solutions on AWS.
- Deep expertise in Apache Kafka and Apache Spark, including architecture, tuning, and deployment.
- Strong background in CI/CD tooling (e.g., Jenkins, GitLab CI, GitHub Actions) and automated testing.
- Solid understanding of Agile methodologies and experience working in iterative development cycles.
- Excellent communication skills and a collaborative mindset.
Skills
awsapache sparkcicdagile