onsite
Lead Data Engineer
Data Engineer
The Lead Data Engineer will be responsible for designing, building, and optimizing data pipelines and ETL workflows in Snowflake, leading data source integrations, and implementing CI/CD practices. This role involves mentoring junior engineers, collaborating with various teams, and ensuring data quality, performance, and governance.
About the role
About the Role
- Lead the architecture and development of data pipelines and ETL workflows in Snowflake utilizing Snowpark, Streams/Tasks, and Snowpipe.
- Design and develop scalable data models that support user 360 views, churn prediction, and recommendation engine inputs.
- Lead the integration of various data sources including MySQL, BigQuery, Redis, Kafka, GCP Storage, and API Gateway.
- Implement CI/CD practices for data pipelines using Git, dbt, and automated testing.
- Define and implement data quality checks and auditing pipelines for both ingestion and transformation layers.
Leadership & Collaboration
- Mentor and guide junior data engineers on data modeling, performance tuning, and Snowflake best practices.
- Collaborate with Data Science, ML, and Backend teams to productionize machine learning features within Snowflake.
- Work closely with Legal, Security, and Infrastructure teams to ensure compliance, privacy, and governance of user data (PII).
- Partner with the Director of Data Platforms and product stakeholders to translate business requirements into technical specifications.
Performance & Scalability
- Tune algorithm performance.
- Establish data partitioning, clustering, and materialized views to optimize query execution.
- Build dashboards and monitors for pipeline health, job success, and data latency metrics, utilizing tools like Looker, Tableau, or Snowsight.
Governance & Best Practices
- Establish and enforce naming conventions, data lineage, and metadata standards across schemas.
- Lead code reviews, enforce documentation standards, and manage schema versioning.
- Contribute to the company’s evolving data mesh and streaming architecture vision.
Skills
SnowflakeSnowparkStreams/TasksSnowpipeEtlMysqlBigQueryRediskafkaGCP StorageAPI GatewayCI/CDGitDbtLookerTableauSnowsight