remote
Senior Data Engineer - Trend Micro
Data Engineer
Senior Data Engineer responsible for designing, building, and scaling data pipelines and platforms using Python, Spark, Airflow, and AWS to enable AI‑driven security analytics and real‑time insights.
About the role
Key Responsibilities
- Design, develop, and maintain scalable ETL pipelines that ingest, transform, and store large volumes of security telemetry.
- Implement real‑time data streaming solutions using Kafka and Spark Structured Streaming.
- Build and optimize data warehouses on AWS (Redshift, S3, Glue) for analytics and machine‑learning workloads.
- Automate workflow orchestration with Apache Airflow, ensuring reliability and observability.
- Collaborate with data scientists, security engineers, and product teams to define data models and deliver actionable insights.
Requirements
- 5+ years of professional experience in data engineering, with strong proficiency in Python and SQL.
- Hands‑on experience with Apache Spark (batch and streaming) and Kafka.
- Deep knowledge of AWS services for data storage, processing, and orchestration (S3, Redshift, Glue, Lambda, Airflow).
- Proven ability to design robust data models and optimize query performance.
- Experience with CI/CD pipelines, containerization (Docker), and version control (Git).
Skills
pythonsqlapache sparkawskafka