remote
BigData DevOps Engineer - Experian
Devops Engineer
Lead the design and automation of scalable Big Data pipelines, leveraging AWS, Spark, Docker, and Kubernetes to deliver reliable, high‑performance data services.
About the role
Key Responsibilities
- Architect, develop, and maintain end‑to‑end Big Data pipelines using Apache Spark on AWS EMR.
- Implement CI/CD workflows with Git, Jenkins, and Terraform to automate deployment and infrastructure provisioning.
- Containerize services with Docker and orchestrate with Kubernetes, ensuring high availability and scalability.
- Monitor performance, troubleshoot issues, and optimize resource utilization across data processing workloads.
- Collaborate with data scientists and application teams to integrate data services into production environments.
Requirements
- 5+ years of experience in Big Data engineering and DevOps practices.
- Proficient with AWS services (EMR, S3, EC2, CloudFormation) and container orchestration.
- Strong scripting skills in Python and Bash.
- Experience with CI/CD pipelines, Git, and infrastructure as code.
- Excellent problem‑solving skills and ability to work in a fast‑paced, collaborative environment.
Skills
awsapache sparkdockerkubernetescicdpython