onsite
DataOps Engineer - Transport for NSW
Data Engineer
Lead end‑to‑end data pipeline development and automation, ensuring high‑quality, scalable data flows across the organization using Python, Airflow, Docker, and AWS services.
About the role
Key Responsibilities
- Design, build, and maintain robust data pipelines that ingest, transform, and deliver data from diverse sources to analytics and reporting platforms.
- Implement CI/CD workflows for data pipelines using Docker, Git, and cloud CI tools, ensuring rapid, reliable deployments.
- Collaborate with data scientists, analysts, and business stakeholders to define data quality standards, monitoring, and alerting mechanisms.
- Optimize pipeline performance and cost on AWS, leveraging services such as S3, Redshift, Glue, and Lambda.
- Document architecture, processes, and best practices to support knowledge transfer and compliance.
Requirements
- 3+ years of experience in data engineering or DataOps roles, with a strong focus on pipeline automation.
- Proficiency in Python, SQL, and orchestration tools like Apache Airflow.
- Hands‑on experience with Docker, Kubernetes, and AWS data services.
- Solid understanding of ETL/ELT concepts, data modeling, and data quality practices.
- Excellent problem‑solving skills and a collaborative mindset.
Skills
pythonsqlairflowdockeraws