onsite
Senior Data Backbone Engineer - New Yorker
Software Engineer
Lead the design and maintenance of scalable data pipelines, ensuring high‑quality data flow across the organization using Python, Spark, Airflow, and AWS services.
About the role
Key Responsibilities
- Architect, develop, and optimize large‑scale data pipelines that ingest, transform, and deliver data to downstream analytics and reporting systems.
- Collaborate with data scientists, product managers, and business stakeholders to define data requirements and ensure data quality and integrity.
- Implement and maintain data models, schemas, and metadata management practices across relational and big‑data platforms.
- Leverage AWS services (Glue, Redshift, S3, EMR) and streaming technologies (Kafka, Kinesis) to build robust, fault‑tolerant data workflows.
- Monitor pipeline performance, troubleshoot issues, and continuously improve throughput and reliability.
Requirements
- 5+ years of experience in data engineering or related roles, with a strong focus on pipeline development.
- Proficiency in Python, SQL, and Apache Spark for data processing.
- Hands‑on experience with Airflow for workflow orchestration and AWS data services.
- Solid understanding of data modeling, ETL best practices, and data governance.
- Excellent problem‑solving skills and ability to work collaboratively in a fast‑paced environment.
Skills
pythonsqlapache sparkawskafka