remote
Lead Data Engineer with AI experience - 3Pillar Global
Data Engineer
Lead Data Engineer driving AI‑native product development, architecting scalable data pipelines on AWS, and enabling machine learning workflows with Python, Spark, and advanced data modeling techniques.
About the role
Key Responsibilities
- Design, build, and maintain end‑to‑end data pipelines that support AI and ML workloads across the HelixAI platform.
- Lead data architecture decisions, ensuring scalability, reliability, and performance on AWS services such as S3, Redshift, and EMR.
- Collaborate with data scientists and product teams to translate business requirements into robust data models and feature stores.
- Implement best practices for data quality, governance, and security, including automated testing and monitoring.
- Mentor and guide a team of data engineers, fostering a culture of continuous improvement and innovation.
Requirements
- 10+ years of experience in data engineering, with a strong focus on AI/ML pipelines.
- Proficiency in Python, SQL, and Apache Spark for large‑scale data processing.
- Deep expertise in AWS data services (S3, Redshift, EMR, Glue, Athena).
- Solid understanding of data modeling, ETL/ELT best practices, and data governance.
- Excellent communication skills and a proven track record of leading technical teams.
Skills
pythonsqlawsapache sparkmachine learning