onsite
Data Engineer II, IAM and Abuse Prevention - Amazon.com
Data Engineer
Data Engineer II focused on building scalable, secure data pipelines and analytics platforms for identity security and abuse prevention, leveraging Python, SQL, Spark, and AWS services.
About the role
Key Responsibilities
- Design, develop, and maintain high‑throughput data pipelines that ingest and process security telemetry, authentication logs, and abuse detection signals.
- Implement data models and transformation logic using Python, SQL, and Apache Spark to enable real‑time and batch analytics.
- Collaborate with security analysts, threat researchers, and product teams to translate investigative requirements into robust data solutions.
- Ensure data security, privacy, and compliance by applying encryption, access controls, and audit logging within AWS environments.
- Optimize pipeline performance and cost efficiency by leveraging AWS services such as S3, Glue, Kinesis, and Redshift.
Requirements
- 3+ years of experience building large‑scale data pipelines and ETL processes in a cloud environment.
- Proficiency in Python and SQL, with hands‑on experience using Apache Spark or similar distributed processing frameworks.
- Strong understanding of AWS data services (S3, Glue, Kinesis, Redshift, Lambda) and best practices for security and governance.
- Experience working with security‑focused datasets, including logs, telemetry, and fraud detection data.
- Ability to troubleshoot complex data issues, write clean production code, and collaborate across cross‑functional teams.
Skills
pythonsqlapache sparkaws