onsite
Data Engineer - Alltech Consulting Services, Inc.
Data Engineer
Data Engineer focused on supporting and maintaining a Hadoop platform, ensuring high availability, performance tuning, and troubleshooting at L3 level. Requires deep knowledge of HDFS, Spark, and Linux system administration.
About the role
Key Responsibilities
- Provide L3 support for the Hadoop Distributed File System (HDFS), diagnosing and resolving performance and reliability issues.
- Monitor cluster health, perform capacity planning, and implement scaling strategies to meet data growth demands.
- Collaborate with data scientists and developers to optimize Spark jobs and ensure efficient data pipelines.
- Implement and maintain backup, recovery, and disaster‑recovery procedures for Hadoop data stores.
- Document troubleshooting steps, create knowledge base articles, and train junior staff on Hadoop best practices.
Requirements
- 3+ years of hands‑on experience with Hadoop ecosystem, including HDFS, YARN, and Spark.
- Strong Linux system administration skills and familiarity with shell scripting.
- Proficiency in SQL and experience querying large datasets.
- Excellent problem‑solving abilities and a proactive approach to system monitoring.
- Effective communication skills for cross‑functional collaboration.