Lead Cloud Platform Engineer Data & Execution Platform - Fidelity Investments

Job Description:

Note: Fidelity is not providing immigration sponsorship for this position.

The Role

We are seeking a hands-on Lead Cloud Platform Engineer to implement, scale, and operate cloud-native infrastructure and services that power large-scale data processing systems. This role focuses on translating defined architectures into production-grade platforms that are reliable, observable, secure, and performant. You will lead the implementation and operation of a modern execution platform built on Apache Spark for distributed compute and an Airflow orchestration layer and DAG execution environment. The ideal candidate brings deep production experience in Spark and Airflow, and excels at troubleshooting, tuning, and operationalizing distributed systems in AWS environments, while leveraging modern developer productivity tools such as AI-assisted coding and LLM-based workflows.

The Expertise and Skills You Bring

Implement and operate cloud-native platform services for distributed data systems

Scale fault-tolerant, high-throughput systems aligned with architectural patterns

Own Spark data pipelines and Airflow orchestration layer and DAG execution

Tune Spark workloads (partitioning, memory, execution plans, shuffle optimization)

Troubleshoot Spark jobs and Airflow DAGs across performance and failures

Operate and optimize Kubernetes-based execution environments, including node group scaling, workload placement, and resource utilization

Troubleshoot Kubernetes infrastructure and workload issues, including scheduling, networking, and runtime performance

Leverage developer productivity tools (e.g., GitHub Copilot, LLMs) to accelerate development, debugging, and operational workflows.

Drive operational excellence including monitoring, incident response, and RCA

Implement observability (metrics, logging, tracing, dashboards, alerting)

Define and manage SLIs/SLOs for platform reliability

Deploy solutions using AWS services (EKS, EC2, S3, Lambda, RDS, etc.) (Implement secure networking (VPCs, IAM, subnets, load balancing)

Maintain CI/CD pipelines and deployment automation

Lead execution across planning, delivery, and cross-team coordination

Mentor engineers and promote reliability and scalability best practices

Strong understanding of distributed systems (fault tolerance, scalability, consistency

Expertise in Apache Spark (tuning, debugging, optimization)

Expertise in Apache Airflow (DAG execution, orchestration, troubleshooting)

Strong experience operating Kubernetes (EKS preferred) including cluster scaling and lifecycle management

Hands-on management of node groups, autoscaling, and capacity planning

Deep understanding of Kubernetes networking and security (security groups, network policies, ingress/egress)