onsite
Lead Data Engineer - ANZ
Data Engineer
Lead Data Engineer driving enterprise‑scale data pipelines and lakehouse architecture using Python, Spark, and AWS to empower institutional analytics across credit, risk, and ESG domains.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and lakehouse solutions that support institutional analytics and reporting.
- Lead a team of data engineers in implementing best practices for data ingestion, transformation, and quality assurance.
- Collaborate with data scientists, product owners, and business stakeholders to translate analytical requirements into robust data models.
- Optimize performance of Spark jobs and SQL queries, ensuring high throughput and low latency.
- Implement security, governance, and compliance controls across the data platform.
Requirements
- 5+ years of experience in data engineering with a strong focus on Python, SQL, and Spark.
- Proven expertise in AWS services such as S3, Glue, Redshift, and Lake Formation.
- Hands‑on experience building and managing data lakes, data warehouses, and ETL pipelines.
- Strong understanding of data modeling, schema design, and performance tuning.
- Excellent communication skills and ability to mentor junior engineers.
Skills
pythonsqlapache sparkaws