remote
Staff Data Engineer - Payabli
Data Engineer
Lead the design and delivery of scalable data pipelines and platforms, leveraging Python, Spark, and AWS to enable real‑time payment analytics and monetization for a fast‑growing fintech product.
About the role
Key Responsibilities
- Architect, build, and maintain high‑throughput data pipelines that ingest, transform, and store payment transaction data from multiple sources.
- Design and implement data models and warehouses to support analytics, reporting, and machine‑learning use cases.
- Develop and operate orchestration workflows using Airflow, ensuring reliability, monitoring, and alerting for critical data jobs.
- Collaborate with product, engineering, and security teams to define data requirements, enforce data quality, and implement governance policies.
- Optimize performance and cost of data processing on AWS services such as S3, Redshift, and EMR.
- Mentor junior engineers and promote best practices in code quality, testing, and documentation.
Requirements
- 5+ years of professional experience building large‑scale data pipelines in a cloud environment.
- Strong proficiency in Python and SQL, with hands‑on experience in Apache Spark or similar distributed processing frameworks.
- Deep understanding of data orchestration tools (e.g., Airflow) and streaming platforms such as Kafka.
- Extensive experience with AWS data services (S3, Redshift, EMR, Glue) and infrastructure‑as‑code practices.
- Proven ability to design robust data models, enforce data quality, and mentor technical teams.
Skills
pythonsqlapache sparkairflowawskafka