onsite
Senior Apache Spark Technical Lead - HCLTech
Engineering Manager
Senior technical lead responsible for designing, developing, and supporting high‑performance data pipelines on Apache Spark using Scala and Python, while ensuring platform stability, data quality, and seamless collaboration with product and engineering teams.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data ingestion, transformation, and analytics pipelines on the Apache Spark platform using Scala and Python.
- Collaborate with product managers, data engineers, QA, and business SMEs to translate requirements into robust data solutions across the full SDLC.
- Implement data quality, validation, and monitoring frameworks to ensure reliability and performance of production workloads.
- Provide tier‑2/3 support, incident response, and root‑cause analysis for Spark‑based services, driving continuous improvement.
- Mentor junior engineers, enforce coding standards, and lead code‑review processes to uphold best practices.
Requirements
- 5+ years of hands‑on experience with Apache Spark, proficient in both Scala and Python.
- Strong background in data modeling, ETL design, and large‑scale data processing pipelines.
- Experience with performance tuning, debugging, and troubleshooting Spark clusters in production environments.
- Proven ability to work cross‑functionally with product, engineering, and business teams.
- Bachelor’s degree in Computer Science, Engineering, or related field; advanced degree is a plus.
Skills
scalapythonapache spark