onsite
Data Engineer - Amivero
Data Engineer
Data Engineer responsible for designing, building, and maintaining scalable data pipelines and infrastructure using Python, SQL, AWS, Spark, Docker, and Airflow to enable data-driven decision making.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines that ingest, transform, and load data from diverse sources into data lakes and warehouses.
- Implement and optimize ETL processes using Python, SQL, and Apache Spark to ensure high performance and reliability.
- Deploy and manage containerized services with Docker and orchestrate workflows with Airflow, ensuring automated, repeatable data workflows.
- Collaborate with data scientists, analysts, and product teams to understand data requirements and deliver clean, well-documented datasets.
- Monitor pipeline health, troubleshoot issues, and implement monitoring and alerting solutions on AWS.
- Document data models, pipeline logic, and best practices to support knowledge sharing and compliance.
Requirements
- 3+ years of experience in data engineering or related field.
Skills
pythonsqlawsapache sparkdockerairflow