remote
Senior Data Engineer - Macquarie Group
Data Engineer
Senior Data Engineer focused on building scalable data pipelines and analytics platforms to surface critical vulnerability insights, leveraging Python, SQL, Spark, and AWS services.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines that ingest, transform, and store vulnerability data from diverse sources.
- Implement scalable ETL processes using Python, SQL, and Apache Spark to ensure high‑quality, timely data for security analytics.
- Collaborate with detection engineers and security analysts to model data that supports prioritisation and contextualisation of critical vulnerabilities.
- Deploy and optimise data workloads on AWS (e.g., S3, Redshift, Glue, Lambda) ensuring reliability, security, and cost‑effectiveness.
- Monitor pipeline performance, troubleshoot issues, and continuously improve data quality and processing efficiency.
Requirements
- 5+ years of hands‑on experience building data pipelines and ETL solutions in a cloud environment.
- Strong proficiency in Python, SQL, and Apache Spark for large‑scale data processing.
- Deep knowledge of AWS data services (S3, Redshift, Glue, Lambda, etc.) and best practices for security and governance.
- Experience working with security or vulnerability data, understanding of risk scoring and prioritisation concepts.
- Excellent problem‑solving skills, ability to work cross‑functionally, and strong communication of technical concepts to non‑technical stakeholders.
Skills
pythonsqlapache sparkaws