onsite
Software Engineer, Data Engineering
Software Engineer
Senior data engineer building scalable pipelines with Airflow, Beam, Flink, Kafka, and Spark to ingest, transform, and analyze large datasets for real‑time and batch processing.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines using Apache Airflow, Beam, Flink, Kafka, and Spark.
- Implement robust ETL processes for batch and streaming data, ensuring data quality and reliability.
- Collaborate with data scientists and product teams to translate business requirements into scalable data solutions.
- Optimize pipeline performance, troubleshoot issues, and implement monitoring and alerting.
- Document architecture, code, and best practices for future maintenance and knowledge transfer.
Requirements
- 5+ years of experience in data engineering with a strong focus on distributed processing frameworks.
Skills
apache beamapache flinkapache spark