remote
Lead Data Engineer - UST
Data Engineer
Lead Data Engineer responsible for designing and delivering scalable data pipelines, guiding a team of engineers, and collaborating with product and analytics stakeholders using Python, Spark, Airflow and cloud services.
About the role
Key Responsibilities
- Architect, build, and maintain robust, high‑performance data pipelines and ETL processes across multiple product squads.
- Provide technical leadership, code reviews, and mentorship to a team of data engineers, fostering best practices and standards.
- Collaborate with product managers, data analysts, visualization engineers, and data architects to translate business requirements into scalable data solutions.
- Design and implement data models, streaming solutions, and batch processing workflows using technologies such as Spark, Airflow, and Kafka.
- Ensure data reliability, security, and performance in cloud environments (AWS), including monitoring, alerting, and cost optimization.
Requirements
- 5+ years of hands‑on experience in data engineering, with a strong focus on Python, SQL, and big‑data processing frameworks.
- Proven expertise in building and orchestrating pipelines with Apache Spark and Apache Airflow.
- Solid understanding of cloud platforms (AWS) and services such as S3, Redshift, EMR, and Lambda.
- Experience with streaming technologies like Kafka and designing real‑time data flows.
- Demonstrated ability to lead technical teams, mentor engineers, and communicate effectively with cross‑functional stakeholders.
Skills
pythonsqlapache sparkawskafka