remoteonsite
Staff Software Engineer L6 - Databricks
Software Engineer
Lead the design and delivery of large‑scale data and AI services on a cloud‑native platform, driving performance, reliability, and scalability for millions of virtual machines.
About the role
Key Responsibilities
- Architect and implement high‑throughput, fault‑tolerant data pipelines and services using Python, Scala, and Apache Spark.
- Own end‑to‑end delivery of features from design through production, ensuring code quality, performance, and security.
- Collaborate with cross‑functional teams to define product requirements and translate them into scalable, cloud‑native solutions on AWS.
- Mentor and coach junior engineers, fostering a culture of continuous improvement and technical excellence.
- Drive automation of deployment, monitoring, and observability using Kubernetes, CI/CD pipelines, and cloud‑native tooling.
Requirements
- 10+ years of software engineering experience with a strong background in distributed data processing.
- Proficiency in Python, Scala, and Spark, with hands‑on experience building production‑grade data services.
- Deep understanding of AWS services (EC2, S3, EMR, EKS) and container orchestration with Kubernetes.
- Experience designing for high availability, scalability, and performance at petabyte scale.
- Excellent communication skills and a proven track record of mentoring and leading engineering teams.
Skills
pythonscalaapache sparkawskubernetescicd