remote
Data Engineer, Smart Factory Solutions - Magna International
Data Engineer
Design and implement data pipelines for smart factory solutions, leveraging Python, SQL, Azure Data Factory, Spark, and real‑time streaming with Kafka.
About the role
Key Responsibilities
- Develop, test, and maintain scalable ETL pipelines that ingest and transform manufacturing data from IoT sensors and production systems.
- Design data models and storage solutions in Azure (Data Lake, Synapse) to support analytics and machine‑learning use cases.
- Implement real‑time data streaming and processing using Apache Kafka and Spark Structured Streaming.
- Collaborate with software, hardware, and analytics teams to define data requirements and ensure data quality and reliability.
- Monitor pipeline performance, troubleshoot issues, and continuously optimize for latency and cost.
Requirements
- 3+ years of experience building data pipelines in a manufacturing or industrial IoT environment.
- Proficiency in Python for data manipulation and automation.
- Strong SQL skills and experience with Azure Data Factory or similar cloud ETL tools.
- Hands‑on experience with Apache Spark (PySpark) and Kafka for batch and streaming workloads.
- Understanding of data modeling, data warehousing concepts, and best practices for data quality and governance.
Skills
pythonsqlapache sparkkafka