onsite
Senior Consultant Data Platforms Engineer - Databricks - EY Consulting GmbH
Software Engineer
Lead the design, implementation, and optimization of large‑scale data platforms on Databricks for financial services clients, driving data quality, performance, and compliance through advanced Spark and Python solutions.
About the role
Key Responsibilities
- Architect and build end‑to‑end data pipelines on Databricks, leveraging Spark, Delta Lake, and Python to ingest, transform, and serve financial data.
- Collaborate with data scientists, business analysts, and stakeholders to translate business requirements into scalable, maintainable data solutions.
- Optimize query performance and resource utilization across AWS services (S3, Redshift, Glue) and Databricks clusters.
- Implement robust data governance, security, and compliance controls in line with financial regulations.
- Mentor junior engineers, conduct code reviews, and promote best practices in data engineering and DevOps.
Requirements
- 5+ years of experience in data engineering, with a strong focus on Databricks and Spark.
- Proficient in Python, SQL, and data modeling for large‑scale datasets.
- Hands‑on experience with AWS data services and Delta Lake architecture.
- Deep understanding of data governance, security, and regulatory compliance in financial services.
- Excellent communication skills and a proven ability to work in cross‑functional teams.
Skills
databricksapache sparkpythonsqlaws