remote
Senior Data Engineer - Trend Micro Inc.
Data Engineer
Senior Data Engineer responsible for designing, building, and scaling data pipelines on cloud platforms, leveraging Python, Spark, and streaming technologies to deliver reliable, high‑performance data solutions for AI‑driven security products.
About the role
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Python, SQL, and Apache Spark on AWS.
- Implement real‑time data ingestion and processing with Kafka and AWS Kinesis.
- Orchestrate workflows and schedule jobs using Apache Airflow, ensuring reliability and observability.
- Collaborate with data scientists and product teams to provide clean, well‑documented data sets for AI security models.
- Optimize data storage, query performance, and cost efficiency across Redshift, S3, and other AWS services.
Requirements
- 5+ years of professional experience building data pipelines in a cloud environment, preferably AWS.
- Strong proficiency in Python, SQL, and big‑data frameworks such as Apache Spark.
- Hands‑on experience with streaming platforms like Kafka and workflow orchestration tools such as Airflow.
- Solid understanding of data modeling, warehousing concepts, and performance tuning.
- Ability to work collaboratively in an agile, cross‑functional team and communicate technical concepts clearly.
Skills
pythonsqlapache sparkawsairflowkafka