remote
Hadoop Big Data Developer - BV Teck
Software Engineer
Senior Hadoop Big Data Developer building scalable data pipelines and analytics solutions using Hadoop ecosystem, Spark, Hive, and Python/Scala on AWS.
About the role
Key Responsibilities
- Design, develop, and maintain large‑scale Hadoop clusters and data pipelines for batch and real‑time processing.
- Implement Spark jobs, Hive queries, and MapReduce tasks to transform and aggregate enterprise data.
- Integrate data sources from relational databases, Kafka streams, and cloud storage into the Hadoop ecosystem.
- Optimize performance and resource utilization across clusters, tuning configurations and job parameters.
- Collaborate with data scientists and analysts to deliver actionable insights and support machine‑learning workflows.
Requirements
- 5+ years of experience with Hadoop, Spark, Hive, and MapReduce.
- Strong programming skills in Python and Scala.
- Hands‑on experience deploying and managing clusters on AWS (EMR, S3, EC2).
- Proficiency in SQL and data modeling for large datasets.
- Excellent problem‑solving skills and ability to work independently in a remote environment.
Skills
hadoophivepythonscalaaws