onsite
Data Engineer - Westpac Group
Data Engineer
Data Engineer building scalable batch and streaming pipelines on AWS, using Python, Spark, Kafka, and SQL to deliver production‑ready data solutions that drive business outcomes.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines across batch and streaming environments.
- Implement data ingestion, transformation, and loading using Python, Spark, and SQL.
- Integrate with Apache Kafka for real‑time data streaming and ensure high availability.
- Collaborate with data architects, engineers, and stakeholders to translate business requirements into robust, production‑ready solutions.
- Optimize performance, monitor pipeline health, and troubleshoot issues in a cloud‑native AWS environment.
Requirements
- Proven experience building data pipelines with Python, Spark, and SQL.
- Hands‑on knowledge of Apache Kafka and streaming data concepts.
- Strong understanding of AWS services (S3, Redshift, Glue, EMR, Lambda).
- Experience with data modeling, ETL design, and performance tuning.
- Excellent problem‑solving skills and ability to work collaboratively in a fast‑paced environment.
Skills
pythonsqlapache sparkaws