onsite
Principal Engineer, Data Infrastructure - Klaviyo
Software Engineer
Lead the design and delivery of a high‑performance data platform that powers real‑time analytics, batch processing, and data governance across the organization, leveraging AWS, Spark, and Airflow to build scalable lakehouse solutions.
About the role
Key Responsibilities
- Architect and evolve a unified data platform that supports streaming, batch compute, and lakehouse storage, ensuring high availability and performance at scale.
- Drive the technical direction for data ingestion, transformation, and governance, collaborating with data scientists, product managers, and engineering teams.
- Implement and maintain robust data pipelines using Python, Apache Spark, and Airflow, optimizing for cost, latency, and reliability on AWS.
- Establish best practices for data quality, lineage, and security, integrating governance frameworks across the data ecosystem.
- Mentor and lead a cross‑functional engineering team, fostering a culture of continuous improvement and technical excellence.
Requirements
- 10+ years of experience building large‑scale data platforms in a cloud environment.
- Deep expertise in streaming and batch processing technologies (Kafka, Kinesis, Spark, Flink).
- Proven track record with lakehouse architectures (Delta Lake, Iceberg) and data governance tools.
- Strong programming skills in Python and experience with orchestration tools like Airflow.
- Excellent communication skills and ability to influence stakeholders across the organization.
Skills
pythonapache sparkawsairflow