Senior Data QA Engineer – Big Data & Azure Databricks
Experience: 5–10 Years Location: Remote CTC: Up to ₹22 LPA Position: Senior Engineer Environment: 60% On-Premise Big Data | 40% Azure Cloud
Role Overview
We are looking for an experienced Senior Data QA Engineer / Big Data Test Engineer with strong hands-on expertise in Hadoop, Hive, PySpark, Advanced SQL, Python, and Azure Databricks .
The role involves validating large-scale data pipelines, performing source-to-target reconciliation, testing ETL/data transformations, and automating data validation across on-premise Hadoop ecosystems and Azure cloud environments .
Key Responsibilities
- Perform end-to-end ETL/Data Warehouse and Big Data testing across source, transformation, and target layers.
- Validate large datasets within Hadoop/HDFS , including CSV and Parquet files.
- Write complex HiveQL and Advanced SQL queries for data validation, reconciliation, and transformation testing.
- Use PySpark/Spark for large-scale data processing and validation.
- Perform data testing and validation within Azure Databricks .
- Navigate Databricks workspace, manage files using commands such as %fs, and work with Unity Catalog .
- Validate data movement and consistency between on-prem Hadoop/Hive and Azure/cloud environments .
- Develop Python-based automation scripts for data validation and reconciliation.
- Perform source-to-target, record-level, aggregate-level, and apple-to-apple data comparison .
- Work with Snowflake or similar cloud data warehouse platforms.
- Use UNIX/Linux shell scripting for file validation, log analysis, and ETL testing activities.
- Monitor and validate scheduled ETL/data pipeline jobs using Control-M .
- Work with database environments including SQL Server/SSMS .
- Track requirements, user stories, defects, and test activities using Jira and Confluence .
Must-Have Skills
- Hadoop / HDFS – Strong hands-on experience
- Apache Hive / HiveQL – Strong hands-on experience
- PySpark / Apache Spark – Strong hands-on experience
- Advanced SQL – Complex queries and data validation
- Python – Strong scripting/automation experience; critical skill
- ETL / Data Warehouse Testing – Strong practical experience
- Azure Databricks – Mandatory
- Snowflake – Important
- Strong source-to-target data validation and reconciliation experience
Good-to-Have Skills
- Unity Catalog
- Databricks %fs and file management
- Python data-comparison/reconciliation libraries such as DataComPy
- Control-M
- UNIX/Linux Shell Scripting
- SQL Server / SSMS
- PyCharm
- Jira & Confluence
- Cross-platform Hadoop–Databricks connectivity
- Hive Metastore integration
Pay: ₹1,000,000.00 - ₹2,200,000.00 per year
Benefits: