onsite
Senior Lead Software Engineer Python, Scala, Spark, AWS - Capital One
Software Engineer
Lead backend engineering initiatives, architecting scalable data pipelines with Python, Scala, and Spark on AWS to solve complex business challenges in a fast‑paced, collaborative environment.
About the role
Key Responsibilities
- Architect and lead the development of large‑scale data processing pipelines using Python, Scala, and Apache Spark on AWS.
- Mentor and guide a team of backend engineers, fostering best practices in code quality, testing, and CI/CD.
- Collaborate with product, data science, and infrastructure teams to translate business requirements into robust, scalable solutions.
- Drive performance optimization, cost efficiency, and reliability across distributed systems.
- Champion continuous improvement, staying current with emerging technologies and industry trends.
Requirements
- 10+ years of software engineering experience with a strong focus on backend development.
- Proven expertise in Python, Scala, and Apache Spark for large‑scale data processing.
- Deep knowledge of AWS services (EMR, S3, Lambda, Glue, Redshift, etc.) and cloud architecture best practices.
- Experience leading technical teams and delivering complex projects on time.
- Excellent communication skills and a collaborative mindset.
Skills
pythonscalaapache sparkaws