remote

Lead Data Engineer - T&T - McCormick & Company

Data Engineer

We're looking for a Data Engineer focused on designing and building scalable technical solutions. This lead role requires 8+ years of relevant experience.

About the role

Competitive compensation
Career growth opportunities
Flexibility and Support for Diverse Life Stages and Choices
Partner with data product managers to gather and deliver data pipelines.
Design ETL solutions including data quality, data security, and data pipeline resiliency.
Execute ETL solutions including data security, data quality and performance requirements.
Design, build and implement ELT pipelines to efficiently ingest and transform data from a wide variety of data sources and deliver datasets that meet business requirements.
Optimize performance for large datasets and data workflows for performance, scalability, and reliability to support business needs.
Develop and maintain scalable data pipelines leveraging Azure Synapse, PySpark, APIs, and SQL & performing advanced data cleaning, transformation, and manipulation to ensure high-quality, and reliable data flows.
Implement CI/CD processes to streamline and automate data pipelines deployment
Apply data validation frameworks (Great Expectations, Fabric-native tools) to maintain accuracy
Utilize partitioning, indexing, clustering strategies to enhance query performance
Collaborate with Data Science, AI, and Data product teams to optimize performance and cost effectiveness of their solutions.
Identify and support the design of internal process improvements, including automating manual processes, optimizing data product delivery, and redesigning solutions for enhanced scalability.
Implement solution adjustments to improve performance and cost-effectiveness of data products.
Monitor and troubleshoot the data pipelines proactively, which includes leading the support of data-related product pipeline issues to resolve data errors.
Provide expert-level support and guidance to data teams across the Enterprise.

Skills

sql

About the role

Competitive compensation
Career growth opportunities
Flexibility and Support for Diverse Life Stages and Choices
Partner with data product managers to gather and deliver data pipelines.
Design ETL solutions including data quality, data security, and data pipeline resiliency.
Execute ETL solutions including data security, data quality and performance requirements.
Design, build and implement ELT pipelines to efficiently ingest and transform data from a wide variety of data sources and deliver datasets that meet business requirements.
Optimize performance for large datasets and data workflows for performance, scalability, and reliability to support business needs.
Develop and maintain scalable data pipelines leveraging Azure Synapse, PySpark, APIs, and SQL & performing advanced data cleaning, transformation, and manipulation to ensure high-quality, and reliable data flows.
Implement CI/CD processes to streamline and automate data pipelines deployment
Apply data validation frameworks (Great Expectations, Fabric-native tools) to maintain accuracy
Utilize partitioning, indexing, clustering strategies to enhance query performance
Collaborate with Data Science, AI, and Data product teams to optimize performance and cost effectiveness of their solutions.
Identify and support the design of internal process improvements, including automating manual processes, optimizing data product delivery, and redesigning solutions for enhanced scalability.
Implement solution adjustments to improve performance and cost-effectiveness of data products.
Monitor and troubleshoot the data pipelines proactively, which includes leading the support of data-related product pipeline issues to resolve data errors.
Provide expert-level support and guidance to data teams across the Enterprise.