remote
Senior Data Engineer - T&T - McCormick & Company
Data Engineer
We're looking for a Data Engineer focused on designing and building scalable technical solutions. This senior role requires 6+ years of relevant experience.
About the role
- Competitive compensation
- Career growth opportunities
- Flexibility and Support for Diverse Life Stages and Choices
- We prioritize our communities and the planet we share. We are proud to be awarded as a Diversity Inc. Top 50 company for Diversity and have multiple Sustainability awards (ranking #22 in the World and #1 in Food Products)
- Wellbeing programs including Physical, Mental and Financial wellness
- Tuition assistance
- Partner with data product managers to gather and deliver data pipelines.
- Design ETL solutions including data quality, data security, and data pipeline resiliency.
- Execute ETL solutions including data security, data quality and performance requirements.
- Design, build and implement ELT pipelines to efficiently ingest and transform data from a wide variety of data sources and deliver datasets that meet business requirements.
- Optimize performance for large datasets and data workflows for performance, scalability, and reliability to support business needs.
- Develop and maintain scalable data pipelines leveraging Azure Synapse, PySpark, APIs, and SQL & performing advanced data cleaning, transformation, and manipulation to ensure high-quality, and reliable data flows.
- Implement CI/CD processes to streamline and automate data pipelines deployment
- Apply data validation frameworks (Great Expectations, Fabric-native tools) to maintain accuracy
- Utilize partitioning, indexing, clustering strategies to enhance query performance
- Collaborate with Data Science, AI, and Data product teams to optimize performance and cost effectiveness of their solutions.
- Identify and support the design of internal process improvements, including automating manual processes, optimizing data product delivery, and redesigning solutions for enhanced scalability.
- Implement solution adjustments to improve performance and cost-effectiveness of data products.
- Monitor and troubleshoot the data pipelines proactively, which includes leading the support of data-related product pipeline issues to resolve data errors.
- Provide expert-level support and guidance to data teams across the Enterprise.
Skills
sqlazuredatabricksgreat expectationsprocess improvement