Role Overview
A leading technology consulting organization is seeking an experienced Data Engineering Consultant with deep expertise in the Microsoft Azure ecosystem. In this role, you will be responsible for architecting, developing, and sustaining scalable data frameworks that support enterprise-wide data needs. You will work across the full data lifecycle — from ingestion and transformation through to governance and performance optimization — enabling clients to derive value from their data assets reliably and efficiently.
Key Responsibilities
- Design, build, and maintain scalable, end-to-end data solutions on Microsoft Azure tailored to business data requirements.
- Develop and manage data pipelines for ingesting, transforming, and integrating structured and unstructured data from diverse sources using Apache Spark.
- Implement robust data security, governance, and privacy controls across Azure data platforms in line with organizational standards.
- Monitor, troubleshoot, and continuously optimize data workflows to ensure high reliability, performance, and cost efficiency.
- Apply Azure best practices to optimize data storage and processing solutions across the platform.
- Collaborate with stakeholders to understand data requirements and translate them into technically sound, scalable architectures.
Mandatory Technical & Functional Skills
- Bachelor's degree in Computer Science, a related field, or equivalent practical experience with a minimum of 4+ years in the domain.
- At least 3 years of consulting or client-facing service delivery experience in Azure-based Microsoft data engineering.
- Minimum 1 year of hands-on experience building data ingestion, processing, and analytics pipelines using SQL Server, Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric.
- Practical experience with ETL and data processing using Azure Data Factory, Azure Functions, Azure Data Lake Storage (ADLS), OneLake, and Microsoft Fabric services.
- At least 1 year working with Azure and Big Data technologies including Microsoft Fabric, Databricks, Python, SQL, ADLS/Blob Storage, Spark, and Spark SQL.
- Minimum 1 year of hands-on experience with RDBMS technologies.
- Familiarity with big data file formats and compression techniques.
- Proficiency with developer tools including Azure DevOps, Git, and Visual Studio Team Services (VSTS).
Preferred Technical & Functional Skills
- Experience designing and deploying ETL solutions on Azure Cloud using Azure Data Factory, Synapse Analytics, Azure Functions, and Notebooks.
- Hands-on experience with Microsoft Fabric, including Medallion Architecture, metadata-driven ingestion frameworks, data quality frameworks, and SCD Type 1 & Type 2 transformations via Fabric Notebooks or Azure Databricks.
- Expos