onsite
Big Data Engineer - FERCHAU GmbH Niederlassung Hamburg-City
Data Engineer
Design, build, and maintain scalable big data pipelines using Hadoop, Spark, and Python, ensuring high data quality and performance across distributed environments.
About the role
Key Responsibilities
- Architect and develop large-scale data processing pipelines with Hadoop and Spark.
- Implement data ingestion, transformation, and storage solutions using Python and SQL.
- Integrate streaming data sources with Kafka and manage real‑time data flows.
- Optimize performance and resource utilization on AWS cloud platforms.
- Collaborate with data scientists and analysts to deliver actionable insights.
Requirements
- Proven experience with Hadoop ecosystem (HDFS, YARN, Hive).
- Strong programming skills in Python and SQL.
- Hands‑on knowledge of Spark (PySpark) and Kafka.
- Experience deploying and managing big data workloads on AWS (EMR, S3, Redshift).
- Excellent problem‑solving skills and ability to work in a fast‑paced environment.
Skills
hadooppythonsqlawskafka