remote
Data Engineer, Notifications - Whatnot
Data Engineer
Build and maintain scalable notification pipelines on AWS, using Python, Kafka, and Airflow to deliver real‑time alerts to millions of users while ensuring data quality and performance.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end notification data pipelines using Python, Kafka, and AWS services.
- Implement data quality checks, monitoring, and alerting to ensure high reliability of notification flows.
- Collaborate with product and engineering teams to define data models and schema for notification events.
- Optimize pipeline performance and cost through efficient use of AWS resources and batch processing.
- Document architecture, processes, and best practices for the data engineering team.
Requirements
- 3+ years of experience building data pipelines in a cloud environment.
- Strong proficiency in Python, SQL, and experience with Kafka or similar streaming platforms.
- Hands‑on experience with AWS services (S3, Redshift, Glue, Lambda, Step Functions).
- Familiarity with Airflow or other workflow orchestration tools.
- Excellent problem‑solving skills and a passion for data quality and performance.
Skills
pythonsqlawskafkaairflow