onsite
Senior Lead Data Engineer - Brillio LLC
Data Engineer
Lead end‑to‑end data engineering initiatives, architecting scalable pipelines on AWS, leveraging Spark and Airflow to deliver high‑quality data products for enterprise clients.
About the role
Key Responsibilities
- Design, build, and maintain large‑scale data pipelines using Python, Apache Spark, and SQL across AWS services (S3, Redshift, Glue).
- Lead a cross‑functional team of data engineers, providing mentorship, code reviews, and best‑practice guidance.
- Implement and manage Airflow DAGs for orchestrating batch and streaming workflows, ensuring reliability and observability.
- Collaborate with data scientists and business stakeholders to translate analytical requirements into robust data solutions.
- Optimize performance, cost, and scalability of data infrastructure, applying advanced techniques such as partitioning, caching, and query tuning.
Requirements
- 10+ years of data engineering experience with a proven track record in large‑scale, cloud‑native environments.
- Expertise in Python, Apache Spark, SQL, and AWS data services (Glue, Redshift, Athena).
- Strong experience with Airflow or similar workflow orchestration tools.
- Excellent leadership, communication, and problem‑solving skills.
- Hands‑on experience with data modeling, ETL design, and performance tuning.
Skills
pythonapache sparksqlawsairflow