onsite
Principal Data Engineer - Pilbara Ports Authority
Data Engineer
Lead the design and delivery of scalable data pipelines and analytics platforms, leveraging Python, Spark, and Azure services to support enterprise‑wide reporting and decision‑making.
About the role
Key Responsibilities
- Architect, build, and maintain high‑performance data pipelines and data lake solutions on Azure.
- Design and implement robust ETL processes using Python, SQL, and Apache Spark to ingest, transform, and store large‑scale bulk export data.
- Collaborate with data scientists, analysts, and business stakeholders to define data models, ensure data quality, and enable self‑service analytics.
- Establish best practices for data governance, security, and performance monitoring across the data platform.
- Mentor junior engineers, conduct code reviews, and drive continuous improvement of engineering standards.
Requirements
- 5+ years of hands‑on experience designing and operating cloud‑based data platforms, preferably on Microsoft Azure.
- Strong proficiency in Python, SQL, and Spark for large‑scale data processing.
- Demonstrated expertise in building data warehouses or data lakes using Azure Data Factory, Synapse, or Databricks.
- Solid understanding of data modeling, ETL design patterns, and performance optimization techniques.
- Excellent problem‑solving skills and the ability to communicate complex technical concepts to non‑technical stakeholders.
Skills
pythonsqlapache spark