remoteonsite
Lead Engineer Data Streaming - Empower
Software Engineer
Lead Engineer for Data Streaming driving real‑time data pipelines with Kafka, Spark, and AWS services, architecting scalable solutions and mentoring a high‑performing team.
About the role
Key Responsibilities
- Design, build, and maintain high‑throughput streaming pipelines using Kafka and Spark Streaming.
- Architect and optimize data ingestion, transformation, and delivery workflows on AWS (Kinesis, S3, EMR).
- Lead a small team of engineers, providing technical guidance, code reviews, and mentorship.
- Collaborate with data science and product teams to define data models and streaming use cases.
- Ensure reliability, scalability, and security of streaming infrastructure through monitoring, alerting, and incident response.
Requirements
- 5+ years of experience in data engineering with a focus on real‑time streaming.
- Proficiency in Kafka, Spark, and Python; experience with Scala is a plus.
- Hands‑on experience deploying and managing streaming workloads on AWS.
- Strong problem‑solving skills and ability to work in a fast‑paced, collaborative environment.
- Excellent communication skills and a passion for mentoring junior engineers.
Skills
kafkaapache sparkpythonawsscala