remote
Principal Data Engineer / Associate Director Software Engineering
Data Engineer
Lead the design and implementation of large‑scale data pipelines using Apache Beam, Flink, Kafka, and Spark, driving performance, reliability, and innovation across the organization.
About the role
Key Responsibilities
- Architect and develop end‑to‑end data pipelines that ingest, transform, and analyze terabytes of data in real time.
- Lead a team of data engineers, mentoring best practices in streaming and batch processing.
- Collaborate with data scientists and product managers to translate business requirements into scalable solutions.
- Optimize pipeline performance, ensuring low latency and high throughput across distributed environments.
- Define and enforce data quality, security, and governance standards.
Requirements
- 10+ years of experience in data engineering with deep expertise in Apache Beam, Flink, Kafka, and Spark.
- Proven track record of leading large, cross‑functional teams in a fast‑paced environment.
- Strong knowledge of distributed systems, cloud platforms, and container orchestration.
- Excellent communication skills and ability to translate complex technical concepts to stakeholders.
Skills
apache beamapache flinkapache spark