onsite
Data DevOps Lead / Associate Director, Data and Analytics - HSBC
Software Engineer
Lead data‑focused DevOps initiatives, designing and operating scalable data pipelines and platforms on cloud infrastructure using Python, Spark, AWS, Docker, and Kubernetes.
About the role
Key Responsibilities
- Define and execute the DevOps strategy for data and analytics platforms, ensuring high availability, security, and performance.
- Architect, build, and maintain scalable data pipelines using Python, SQL, and Apache Spark on AWS.
- Implement CI/CD pipelines, containerization (Docker) and orchestration (Kubernetes) to automate deployment and monitoring of data services.
- Collaborate with data engineers, analysts, and business stakeholders to translate requirements into robust, production‑grade solutions.
- Establish governance, monitoring, and incident‑response processes for data infrastructure, driving continuous improvement.
Requirements
- 10+ years of experience in data engineering and DevOps, with a proven track record leading large‑scale data platforms.
- Strong proficiency in Python, SQL, and Apache Spark for data processing.
- Deep hands‑on experience with AWS services (e.g., S3, EMR, Redshift, Lambda) and infrastructure‑as‑code tools such as Terraform.
- Expertise in container technologies (Docker) and orchestration (Kubernetes) for building reproducible, scalable environments.
- Excellent communication and leadership skills, capable of influencing cross‑functional teams and driving technical vision.
Skills
pythonsqlapache sparkawsdockerkubernetescicdterraform