remoteonsite
Senior Software Engineer L5 - Databricks
Software Engineer
Lead the design and implementation of scalable data processing pipelines using Python, Scala, and Spark on AWS, driving performance and reliability for a global data and AI platform.
About the role
Key Responsibilities
- Architect, develop, and maintain high‑throughput data pipelines and services in Python and Scala, leveraging Apache Spark on a distributed cloud environment.
- Collaborate with cross‑functional teams to define feature requirements, estimate effort, and deliver production‑ready code on time.
- Design and implement CI/CD pipelines, automated testing, and monitoring solutions to ensure reliability and rapid deployment.
- Optimize existing workloads for cost, latency, and scalability, applying best practices in distributed systems and cloud architecture.
- Mentor junior engineers, conduct code reviews, and promote a culture of technical excellence and continuous improvement.
Requirements
- 8+ years of software engineering experience with a strong focus on data processing and distributed systems.
- Proficiency in Python and Scala, with hands‑on experience building Spark applications at scale.
- Deep understanding of AWS services (EC2, S3, EMR, EKS) and container orchestration with Kubernetes.
- Experience designing CI/CD pipelines, automated testing, and observability for large‑scale services.
- Excellent problem‑solving skills, strong communication, and a passion for building reliable, high‑performance systems.
Skills
pythonscalaapache sparkawskubernetescicd