remote
Lead Software Engineer - Platform Engineering Databricks - JPMorganChase
Software Engineer
Lead the design and delivery of scalable data platforms on Databricks, driving AI/ML initiatives and ensuring data quality, security, and performance across the enterprise.
About the role
Key Responsibilities
- Architect, develop, and maintain large‑scale data pipelines and analytics workloads on Databricks using Spark, Python, and Scala.
- Collaborate with data scientists and product teams to deploy machine learning models into production, ensuring reproducibility and scalability.
- Implement robust data governance, security, and compliance controls across the data platform.
- Mentor and lead a cross‑functional engineering team, fostering best practices in code quality, CI/CD, and performance tuning.
- Drive continuous improvement of platform reliability, monitoring, and cost optimization in cloud environments.
Requirements
- 10+ years of software engineering experience with a strong focus on data platforms.
- Deep expertise in Databricks, Apache Spark, and distributed data processing.
- Hands‑on experience with cloud services (AWS, Azure, or GCP) and data lake architecture.
- Strong understanding of data governance, security, and compliance best practices.
Skills
databricksapache sparkpythonscalamachine learning