onsite
Data Engineer - abtrace
Data Engineer
Data Engineer responsible for building and maintaining scalable data pipelines using Python, SQL, and Spark on AWS, ensuring data quality and performance across the organization.
About the role
Key Responsibilities
- Design, develop, and maintain robust ETL pipelines to ingest, transform, and load data from diverse sources into data warehouses.
- Optimize data workflows using Spark, SQL, and Python, ensuring high performance and reliability.
- Implement and manage data orchestration with Airflow, scheduling and monitoring batch jobs.
- Collaborate with data scientists and analysts to provide clean, well‑documented datasets for analytics and ML projects.
- Ensure data security, compliance, and governance across all data assets.
Requirements
- 3+ years of experience as a Data Engineer or similar role.
- Hands‑on experience with AWS services (S3, Redshift, EMR, Glue).
- Strong knowledge of containerization (Docker) and workflow orchestration (Airflow).
- Excellent problem‑solving skills and ability to work in a fast‑paced environment.
Skills
pythonsqlapache sparkawsdockerairflow