onsite
Senior Data Engineer - eSC or eDV Required - IBM
Data Engineer
Senior Data Engineer driving hybrid cloud and AI initiatives, building scalable data pipelines and lakehouse architectures using Python, Spark, SQL, and AWS services.
About the role
Key Responsibilities
- Design, develop, and maintain large‑scale data pipelines and lakehouse solutions on AWS, leveraging Spark, SQL, and Python.
- Implement robust ETL processes, data quality checks, and performance tuning for high‑volume datasets.
- Collaborate with data scientists, architects, and business stakeholders to translate analytical requirements into production‑ready workflows.
- Ensure data governance, security, and compliance across hybrid cloud environments.
- Mentor junior engineers and contribute to best‑practice documentation and tooling.
Requirements
- 5+ years of experience in data engineering with a strong focus on cloud platforms.
- Proficiency in Python, SQL, and Apache Spark for data processing.
- Hands‑on experience with AWS services (S3, Redshift, Glue, EMR, Athena).
- Solid understanding of Hadoop ecosystem, data lake architecture, and ETL best practices.
- Excellent problem‑solving skills and ability to work in a fast‑paced, collaborative environment.
Skills
pythonsqlapache sparkawshadoop