remote
Senior Big Data Engineer - SAP
Data Engineer
Senior Big Data Engineer responsible for designing, building, and optimizing large‑scale data pipelines and analytics platforms using Hadoop, Spark, Kafka, and cloud services to support enterprise‑grade solutions.
About the role
Key Responsibilities
- Design, develop, and maintain high‑performance data pipelines on Hadoop and Spark clusters.
- Implement real‑time streaming solutions with Kafka to ingest and process large volumes of transactional data.
- Collaborate with data scientists and product teams to create scalable data models and enable advanced analytics.
- Optimize data storage, query performance, and cost efficiency on AWS services such as S3, EMR, and Redshift.
- Establish best practices for data quality, governance, and security across the data platform.
Requirements
- 5+ years of hands‑on experience with Hadoop ecosystem tools and Spark programming.
- Proficiency in Scala or Python for data processing and pipeline development.
- Strong knowledge of Kafka, AWS cloud services, and SQL/NoSQL databases.
- Experience designing data models and implementing ETL/ELT workflows at scale.
- Excellent problem‑solving skills and ability to work in an agile, cross‑functional environment.
Skills
apache sparkkafkascalapythonawssql