onsite
Infrastructure Engineer Senior - Data and Automation Red Hat OpenShift, Apache Spark - PNC Financial Services Group
Devops Engineer
Senior Infrastructure Engineer focused on data and automation, building containerized Spark workloads on OpenShift, managing Iceberg data lakes, and delivering robust CI/CD pipelines with Docker and Kubernetes.
About the role
Key Responsibilities
- Design, implement, and maintain containerized Apache Spark applications on Red Hat OpenShift, ensuring high availability and scalability.
- Develop and manage automated delivery pipelines using CI/CD tools, integrating code quality checks, security scanning, and deployment to production clusters.
- Collaborate with data engineering teams to optimize Apache Iceberg table performance, schema evolution, and data lifecycle policies.
- Provision and configure Kubernetes resources, including custom operators, Helm charts, and service meshes, to support data workloads.
- Monitor cluster health, troubleshoot performance bottlenecks, and implement capacity planning for compute and storage resources.
Requirements
- 5+ years of experience in DevOps or infrastructure engineering with a focus on data platforms.
- Proven expertise in Red Hat OpenShift, Docker, and Kubernetes orchestration.
- Hands‑on experience with Apache Spark and Iceberg data lake architecture.
- Strong scripting skills in Python or Bash for automation and tooling.
- Familiarity with CI/CD pipelines (Jenkins, GitLab CI, ArgoCD) and cloud‑native monitoring (Prometheus, Grafana).
Skills
apache sparkdockercicdkubernetes