onsite
Senior Data Engineer - Palantir Foundry Hybrid, Canada - Unit8 SA
Data Engineer
Senior Data Engineer leading large‑scale data pipelines on Palantir Foundry, building robust ETL workflows, data models, and cloud‑native solutions using Python, Spark, and AWS to power AI initiatives across diverse industries.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines on Palantir Foundry, ensuring high data quality and performance.
- Implement complex ETL processes using Python, SQL, and Apache Spark to ingest, transform, and load data from heterogeneous sources.
- Collaborate with data scientists and business stakeholders to model data structures that support advanced analytics and AI workloads.
- Leverage AWS services (S3, Glue, Redshift, Lambda) to build a resilient data lake architecture and automate data workflows.
- Monitor pipeline health, troubleshoot issues, and optimize resource usage for cost efficiency.
- Document data lineage, metadata, and best practices to enable governance and compliance.
Requirements
- 5+ years of experience in data engineering, with a strong background in Python, SQL, and Spark.
- Proven expertise in building and managing data pipelines on cloud platforms, preferably AWS.
- Hands‑on experience with Palantir Foundry or similar data integration platforms.
- Solid understanding of data modeling, ETL design patterns, and data lake concepts.
- Excellent problem‑solving skills and ability to work collaboratively in a fast‑paced environment.
Skills
pythonsqlapache sparkaws