onsite
Lead Software Engineer - Platform Engineering Databricks - JPMorgan Chase Bank, N.A.
Software Engineer
Lead a platform engineering team building scalable, secure data pipelines on Databricks, leveraging Python, Scala, Spark, and AWS to deliver high‑performance analytics solutions.
About the role
Key Responsibilities
- Design, develop, and maintain large‑scale data processing platforms on Databricks using Python, Scala, and Apache Spark.
- Architect and implement cloud‑native solutions on AWS, including storage, compute, and networking components.
- Lead the adoption of containerization and orchestration with Kubernetes to ensure reliable, reproducible deployments.
- Establish CI/CD pipelines, automated testing, and monitoring to guarantee high availability and performance.
- Mentor senior engineers, enforce coding standards, and drive best practices for data security and governance.
Requirements
- 5+ years of professional software engineering experience, with at least 2 years leading platform teams.
- Deep expertise in Python and Scala for data engineering workloads.
- Strong hands‑on experience with Apache Spark and Databricks runtime.
- Proven track record building and operating services on AWS, including IAM, S3, EMR, and Lambda.
- Experience with Kubernetes, Docker, and modern CI/CD tools (e.g., Jenkins, GitLab CI, or GitHub Actions).
Skills
pythonscalaapache sparkdatabricksawskubernetescicd