onsite
Senior Big Data Engineer - Lehmann + Pioneers Digital GmbH
Data Engineer
Lead the design, development, and maintenance of large-scale data pipelines and analytics platforms using Spark, Hadoop, and cloud services, ensuring high performance, reliability, and scalability.
About the role
Key Responsibilities
- Architect and implement end‑to‑end data pipelines for ingesting, processing, and storing terabyte‑scale datasets.
- Optimize Spark and Hadoop jobs for performance, cost, and resource utilization on AWS.
- Design and maintain data models, schemas, and metadata management for analytics and reporting.
- Collaborate with data scientists and product teams to deliver actionable insights and support ML workflows.
- Implement monitoring, alerting, and automated testing for data pipelines and infrastructure.
Requirements
- 5+ years of experience in big data engineering with hands‑on expertise in Spark, Hadoop, and Scala.
- Strong proficiency in Python, SQL, and data modeling.
- Experience deploying and managing data platforms on AWS (EMR, S3, Glue, Redshift).
- Knowledge of streaming technologies such as Kafka or Flink.
- Excellent problem‑solving skills and a proactive, collaborative mindset.
Skills
apache sparkscalapythonsqlawskafka