remoteonsite
Test Engineer - Corporate Lakehouse - Persistent Systems
QA Engineer
Experienced Test Engineer focused on validating and automating data pipelines for a corporate lakehouse environment, leveraging Python, SQL, Spark, and Azure services to ensure data quality, performance, and reliability.
About the role
Key Responsibilities
- Design, develop, and maintain automated test frameworks for data ingestion, transformation, and storage pipelines in a lakehouse architecture.
- Validate data quality, integrity, and performance using Python, SQL, and Spark-based test suites.
- Collaborate with data engineers, architects, and product owners to define test criteria and acceptance criteria for new features.
- Implement CI/CD pipelines to integrate automated tests into the deployment workflow on Azure.
- Monitor, troubleshoot, and resolve data pipeline defects and performance bottlenecks.
Requirements
- 3+ years of experience in test engineering or quality assurance for data platforms.
- Strong proficiency in Python and SQL for data validation and test automation.
- Hands‑on experience with Apache Spark and lakehouse concepts (e.g., Delta Lake, Snowflake).
- Familiarity with Azure services such as Data Lake Storage, Databricks, and DevOps pipelines.
- Experience building CI/CD pipelines and using version control (Git) in an agile environment.
Skills
pythonsqlapache sparkazurecicd