remote
Kafka Engineer - BV Teck
Software Engineer
Senior Kafka Engineer responsible for designing, building, and maintaining high‑throughput streaming pipelines using Kafka, Java, and Python, while leveraging Spark, AWS, Docker, and Kubernetes for scalable, resilient data architectures.
About the role
Key Responsibilities
- Design, develop, and deploy Kafka-based streaming solutions that support real‑time data ingestion and processing.
- Implement and maintain Kafka Connect, Kafka Streams, and KSQL pipelines using Java and Python.
- Integrate streaming data with downstream systems (data lakes, warehouses, analytics platforms) via Spark and other processing frameworks.
- Ensure high availability, fault tolerance, and performance tuning of Kafka clusters in AWS environments.
- Containerize services with Docker and orchestrate deployments using Kubernetes.
- Collaborate with data scientists, DevOps, and product teams to translate business requirements into scalable streaming architectures.
Requirements
- 5+ years of experience with Apache Kafka, including cluster administration and stream processing.
- Strong programming skills in Java and Python, with familiarity in Scala.
- Hands‑on experience with Apache Spark, AWS services (EKS, S3, Kinesis), Docker, and Kubernetes.
- Proven ability to design and optimize large‑scale data pipelines and troubleshoot performance issues.
- Excellent communication skills and a collaborative mindset for remote teamwork.
Skills
kafkajavapythonapache sparkawsdockerkubernetes