remote
Software Engineer - Procloz PTY LTD
Software Engineer
Software Engineer focused on building scalable, reliable data streaming pipelines using Python, Java, Kafka, and Spark on AWS, with containerization via Docker and orchestration in Kubernetes.
About the role
Key Responsibilities
- Design, develop, and maintain distributed streaming frameworks that ingest data from diverse sources into the platform’s database.
- Implement high‑availability, fault‑tolerant pipelines using Kafka, Spark, and other stream processing technologies.
- Collaborate with data scientists and product teams to define ingestion requirements and optimize data flow performance.
- Write clean, well‑tested code, conduct code reviews, and contribute to continuous integration and deployment pipelines.
- Monitor, troubleshoot, and tune production streaming services for scalability and reliability.
Requirements
- Strong experience with Python and Java for building data pipelines.
- Hands‑on expertise with Kafka, Apache Spark, and related stream processing frameworks.
- Proficiency in AWS services (Kinesis, S3, EC2, Lambda) and containerization with Docker.
- Knowledge of Kubernetes for orchestrating microservices at scale.
- Excellent problem‑solving skills and a passion for clean, maintainable code.
Skills
pythonjavakafkaapache sparkawsdockerkubernetes