remote
Senior Data Pipeline Engineer - Netbuilder
Software Engineer
Senior Data Pipeline Engineer leading end‑to‑end design, deployment and operation of observability telemetry pipelines using Python, Spark, Airflow, AWS, Kafka and Docker to deliver scalable, reliable data flows for enterprise clients.
About the role
Key Responsibilities
- Architect, build and maintain large‑scale telemetry pipelines that ingest, transform and route observability data across cloud and on‑prem environments.
- Implement data ingestion solutions with Apache Kafka, stream processing with Apache Spark, and orchestration via Apache Airflow.
- Deploy and manage pipeline components on AWS using services such as S3, EMR, Lambda, and ECS, ensuring high availability and cost efficiency.
- Containerize services with Docker and orchestrate with Kubernetes or ECS, applying best practices for CI/CD and automated testing.
- Collaborate with cross‑functional teams to define data quality, security, and compliance requirements, and enforce them through monitoring and alerting.
Requirements
- 5+ years of experience designing and operating data pipelines in production environments.
- Proficiency in Python, Spark, Airflow, Kafka, and AWS services.
- Hands‑on experience with Docker, Kubernetes, and Terraform for infrastructure as code.
- Strong understanding of observability concepts, metrics, logs, and tracing.
- Excellent communication skills and ability to work independently in client‑facing roles.
Skills
pythonapache sparkawskafkadocker