onsite
Lead Data Engineer
Data Engineer
The Lead Data Engineer will design, build, and optimize data pipelines and ETL workflows in Snowflake, integrating various data sources. This role involves developing scalable data models, leading CI/CD implementations for data pipelines, and ensuring data quality. The engineer will also mentor junior team members, collaborate with cross-functional teams, and establish data governance and best practices.
About the role
Architecture & Development
- Design, build, and optimize data pipelines and ETL workflows in Snowflake using Snowpark, Streams/Tasks, and Snowpipe.
- Develop scalable data models, Algorithm supporting user 360 views, churn prediction, and recommendation engine inputs.
- Lead integration across data sources: MySQL, BigQuery, Redis, Kafka, GCP Storage, and API Gateway.
- Implement CI/CD for data pipelines using Git, dbt, and automated testing.
- Define data quality checks and auditing pipelines for ingestion and transformation layers.
Leadership & Collaboration
- Mentor and guide junior data engineers on data modeling, performance tuning, and Snowflake best practices.
- Partner with Data Science, ML, and Backend teams to productionize machine learning features in Snowflake.
- Work closely with Legal, Security, and Infrastructure teams to ensure compliance, privacy, and governance of user data (PII).
- Collaborate with the Director of Data Platforms and product stakeholders to translate business requirements into technical specifications.
Performance & Scalability
- Tune algorithm performance.
- Establish data partitioning, clustering, and materialized views for fast query execution.
- Build dashboards and monitors for pipeline health, job success, and data latency metrics (e.g., via Looker, Tableau, or Snowsight).
Governance & Best Practices
- Establish and enforce naming conventions, data lineage, and metadata standards across schemas.
- Lead code reviews, enforce documentation standards, and manage schema versioning.
- Contribute to the company’s evolving data mesh and streaming architecture vision.