onsite
Data Engineer - Kompetenzzentrum Das virtuelle Fahrzeug ForschungsgmbH
Data Engineer
Build and maintain scalable data pipelines using Python, SQL, and AWS services, leveraging Spark for large‑scale processing and ensuring robust data modeling and ETL workflows.
About the role
Key Responsibilities
- Design, develop, and optimize data pipelines for ingestion, transformation, and storage across cloud and on‑premise environments.
- Implement ETL processes using Python, SQL, and Apache Spark to support analytics and machine learning initiatives.
- Collaborate with data scientists and product teams to define data models, schemas, and quality metrics.
- Monitor pipeline performance, troubleshoot issues, and implement automated alerts and scaling strategies.
- Maintain documentation of data flows, architecture decisions, and best practices.
Requirements
- 3+ years of experience as a data engineer or similar role.
- Experience designing data models and building ETL pipelines for large datasets.
- Excellent problem‑solving skills and ability to work in a collaborative, fast‑paced environment.
Skills
pythonsqlawsapache spark