onsite
Lead Data Engineer - 10decoders
Data Engineer
Lead Data Engineer driving cloud‑native, high‑performance data pipelines for analytics and AI/ML across banking, fintech, consulting and SaaS domains, leveraging Python, Spark, Kafka, AWS, Airflow and advanced data modeling.
About the role
Key Responsibilities
- Design, develop and maintain scalable batch and real‑time data pipelines using Python, Spark and Kafka on AWS.
- Architect cloud‑native data platforms, ensuring high availability, performance and security.
- Lead a small team of data engineers, providing mentorship, code reviews and technical guidance.
- Collaborate with data scientists, product managers and business stakeholders to translate requirements into robust data solutions.
- Implement CI/CD, monitoring, and automated testing for data workflows using Airflow and AWS services.
Requirements
- 6–8 years of experience in data engineering with a strong focus on cloud technologies.
- Proficiency in Python, Apache Spark, Apache Kafka, SQL and data modeling best practices.
- Hands‑on experience with AWS services (S3, Redshift, EMR, Glue, Lambda) and orchestration tools like Airflow.
- Solid understanding of data architecture, ETL/ELT processes, and performance tuning.
- Excellent communication skills and a proven ability to lead and collaborate across cross‑functional teams.
Skills
pythonapache sparkawsairflowsql