onsite
Data Engineer HCM - Wenco (a Hitachi Construction Machinery subsidiary)
Data Engineer
Data Engineer for the HCM Digital Solutions team, building and maintaining scalable data pipelines and storage for large time‑series workloads on the LANDCROS Connect Insight platform using Python, Spark, and cloud services.
About the role
Key Responsibilities
- Design, develop, and maintain robust ETL pipelines for high‑volume time‑series data using Python, Apache Spark, and Airflow.
- Implement data ingestion and streaming solutions with Kafka to support real‑time analytics.
- Optimize data storage and processing on AWS services (S3, Redshift/Glue) to ensure performance and cost efficiency.
- Collaborate with business stakeholders and data scientists to translate requirements into scalable data models.
- Monitor pipeline health, enforce data quality standards, and troubleshoot issues in production environments.
Requirements
- 3+ years of experience building data pipelines and working with big data technologies such as Spark and Kafka.
- Proficiency in Python and SQL for data transformation and querying.
- Hands‑on experience with AWS data services (S3, Redshift, Glue, Lambda) and infrastructure‑as‑code concepts.
- Familiarity with workflow orchestration tools like Airflow or similar.
- Strong problem‑solving skills and ability to work cross‑functionally with analysts and engineers.
Skills
pythonsqlapache sparkawsairflowkafka