onsite
Data Analytics Test Engineer 00012460 - Auto & General Insurance
QA Engineer
Data Analytics Test Engineer focused on building and executing rigorous testing for large‑scale data pipelines and analytical models using Python, Spark, and Airflow, ensuring data integrity and reliable insights across the enterprise platform.
About the role
Key Responsibilities
- Design, develop, and maintain automated test suites for data pipelines, models, and analytical solutions using Python and Spark.
- Implement data quality checks and validation rules within Airflow workflows to ensure accuracy and consistency across the enterprise data platform.
- Collaborate with data engineers, analysts, and product owners to translate business requirements into technical test specifications.
- Analyze test results, identify root causes, and work with stakeholders to remediate data quality issues.
- Document testing processes, best practices, and maintain a knowledge base for continuous improvement.
Requirements
- Strong experience with Python, SQL, and Apache Spark for data processing and testing.
- Proficiency in orchestrating workflows with Airflow and automating test pipelines.
- Deep understanding of data quality concepts, data profiling, and validation techniques.
- Experience with big data technologies and enterprise data platforms.
- Excellent analytical, problem‑solving, and communication skills.
Skills
pythonsqlapache sparkairflow