remote
Senior Software Engineer - Data Pipelines - Obsidian Security
Software Engineer
Senior Software Engineer building scalable data pipelines in Python on AWS, orchestrating ETL workflows with Spark and Kubernetes to deliver real‑time security insights for SaaS platforms.
About the role
Key Responsibilities
- Design, develop, and maintain high‑throughput data pipelines ingesting security telemetry from SaaS applications.
- Implement ETL processes using Python, Apache Spark, and AWS Glue to transform raw logs into actionable insights.
- Deploy and manage containerized services on Kubernetes, ensuring zero‑downtime updates and autoscaling.
- Automate infrastructure provisioning with Terraform and monitor pipeline health with CloudWatch and Prometheus.
- Collaborate with data scientists and security analysts to refine data models and improve threat detection accuracy.
Requirements
- 5+ years of software engineering experience focused on data engineering or pipeline development.
- Proficiency in Python, SQL, and experience with Spark or similar big‑data frameworks.
- Hands‑on experience building and deploying pipelines on AWS (S3, Redshift, Glue, Lambda).
- Strong knowledge of Kubernetes, Docker, and CI/CD pipelines.
- Excellent problem‑solving skills and a passion for building reliable, scalable systems.
Skills
pythonawsapache sparkkubernetesterraform