remote
Senior Data Engineer - Enterprise B2B Marketplace - Truelogic
Data Engineer
Senior Data Engineer leading enterprise‑grade data pipelines for a B2B marketplace, leveraging Python, Spark, and AWS to design scalable ETL workflows and maintain robust data warehouses.
About the role
Key Responsibilities
- Design, develop, and maintain large‑scale data pipelines that ingest, transform, and load data from diverse sources into a cloud‑based data warehouse.
- Collaborate with data scientists and product teams to define data models, optimize query performance, and ensure data quality and integrity.
- Implement and manage CI/CD workflows for data infrastructure, including automated testing, deployment, and monitoring.
- Leverage AWS services (Redshift, S3, Glue, Lambda) to build scalable, cost‑effective data solutions.
- Document architecture, processes, and best practices for internal teams and stakeholders.
Requirements
- 5+ years of experience in data engineering, with a strong background in Python and SQL.
- Proficiency with Apache Spark and experience building distributed data pipelines.
- Hands‑on experience with AWS data services (Redshift, Glue, S3, Lambda).
- Solid understanding of data modeling, ETL design, and performance tuning.
- Excellent communication skills and ability to work cross‑functionally in a fast‑paced environment.
Skills
pythonsqlapache sparkaws