onsite
Senior Principal Databricks Platform Engineering Lead - JPMorgan Chase & Co.
Software Engineer
Lead the design and automation of a scalable Databricks platform on AWS, using Python and Terraform, while mentoring engineering teams and driving data‑centric innovation for a global financial institution.
About the role
Key Responsibilities
- Architect, build, and maintain a production‑grade Databricks platform on AWS, ensuring high availability, security, and cost efficiency.
- Develop and manage infrastructure‑as‑code pipelines with Terraform and Python to automate provisioning, configuration, and lifecycle management.
- Collaborate with data scientists, analysts, and product owners to translate business requirements into robust data solutions.
- Mentor and guide multiple engineering squads, establishing best practices for code quality, testing, and CI/CD.
- Drive performance tuning, monitoring, and continuous improvement of data pipelines and workloads.
Requirements
- 10+ years of software engineering experience, with at least 5 years focused on cloud data platforms and large‑scale analytics.
- Deep expertise in Python development and Databricks runtime, including Spark optimization.
- Proven experience designing and operating AWS environments, including networking, security, and storage services.
- Strong proficiency with Terraform (or similar IaC tools) and modern CI/CD workflows.
- Excellent communication and leadership skills, with a track record of mentoring senior engineers.
Skills
pythondatabricksawsterraformcicd