onsite
Data Engineer - PlusAI
Data Engineer
Data Engineer building scalable data pipelines and analytics solutions for AI‑driven autonomous truck software, leveraging Python, Spark, AWS, and Kafka to support real‑time data ingestion and model training.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines that ingest, transform, and store large volumes of sensor and telemetry data for AI model training.
- Implement real‑time streaming solutions using Kafka and Spark Structured Streaming to support live analytics and decision‑making.
- Collaborate with data scientists and ML engineers to optimize data schemas, feature stores, and data quality processes.
- Deploy and manage data infrastructure on AWS (S3, Redshift, Glue, EMR) ensuring high availability and cost efficiency.
- Monitor pipeline performance, troubleshoot issues, and continuously improve data processing workflows.
Requirements
- 3+ years of experience as a Data Engineer or similar role in a fast‑paced environment.
- Hands‑on experience with Kafka or other streaming platforms.
- Excellent problem‑solving skills and ability to work collaboratively across cross‑functional teams.
Skills
pythonapache sparkawskafkasql