remoteonsite
Director, Data Engineer III - Morgan Stanley
Data Engineer
Director-level data engineer leading design and delivery of high‑volume, low‑latency data pipelines for post‑trade operations, leveraging Python, Spark, SQL, and cloud services.
About the role
Key Responsibilities
- Architect, develop, and maintain scalable data pipelines that process tens of millions of trades per day.
- Lead a team of data engineers, providing technical guidance and fostering best practices in data engineering and DevOps.
- Design data models and storage solutions on AWS (S3, Redshift, Glue) to support settlement, regulatory reporting, and corporate actions workflows.
- Implement real‑time streaming solutions using Kafka and Spark Structured Streaming for low‑latency trade and position data.
- Collaborate with business analysts, traders, and risk teams to translate functional requirements into robust data solutions.
Requirements
- 10+ years of experience in data engineering, with at least 3 years in a senior or director role.
- Strong proficiency in Python, SQL, and Apache Spark for batch and streaming workloads.
- Hands‑on experience with AWS services (S3, Redshift, EMR, Glue) and container orchestration (Kubernetes/ECS).
- Deep understanding of financial post‑trade processes such as settlements, confirmations, and regulatory reporting.
- Proven ability to lead technical teams, drive architectural decisions, and deliver production‑grade data platforms at scale.
Skills
pythonsqlapache sparkawskafka