remote
Senior Data Engineer - Dark Wolf Solutions
Data Engineer
Senior Data Engineer responsible for designing, building, and optimizing data pipelines and infrastructure for mission‑critical health analytics, using Python, SQL, AWS, Airflow, Docker, and Kafka while applying DevSecOps and TDD practices.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines that ingest, transform, and store operational medicine data.
- Implement and manage cloud‑based data platforms on AWS, ensuring high availability, security, and cost efficiency.
- Automate workflow orchestration with Apache Airflow, incorporating monitoring, alerting, and retry mechanisms.
- Containerize services using Docker and orchestrate them for consistent deployment across environments.
- Apply DevSecOps and Test‑Driven Development practices to deliver reliable, customer‑centric software solutions.
Requirements
- 5+ years of professional experience in data engineering, with strong proficiency in Python and SQL.
- Hands‑on experience with AWS services (e.g., S3, Redshift, Lambda, Glue) and infrastructure‑as‑code tools.
- Proven expertise in building and managing Airflow DAGs and streaming pipelines using Kafka or similar technologies.
- Solid understanding of containerization (Docker) and CI/CD pipelines within a DevSecOps framework.
- Experience with Test‑Driven Development, code reviews, and automated testing for data pipelines.
Skills
pythonsqlawsdockerkafka