AI QA Analyst - Ryz Labs

At Ryz Labs , we are looking for an AI QA Analyst to join one of our clients, an AI-first travel and hospitality company. In this role, you will act as the bridge between model development and user experience. Your primary goal is to ensure our generative models are accurate, safe, and contextually aware by implementing rigorous evaluation frameworks and data-driven insights.

Qualifications:

• Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content.
• Experience managing, cleaning, or labeling datasets for machine learning or NLP projects.
• Familiarity with the AI lifecycle, Prompt Engineering, and tools like Google Sheets/Excel (SQL or Python is a significant plus).

Key Responsibilities:

• Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH).
• Build and maintain high-quality "Golden Datasets" and benchmarks to stress-test the model across various domains and edge cases.
• Conduct pre-deployment testing and A/B model comparisons to identify performance regressions or improvements.
• Categorize model failures (hallucinations, logic errors, tone drift) to provide actionable feedback to the Engineering and Research teams.
• Help define and refine the rubric for "what a good response looks like" as the product evolves.
• Building LLM-as-a-judge system with golden prompts and automation for output scoring
• Regression monitoring: detecting quality degradation across prompt, model, and config changes
• Evaluating agent routing
• AI Safety & Guardrails checks

Stack: Braintrust / Langsmith

About RYZ Labs:

RYZ Labs is a startup studio founded in 2021 by two lifelong entrepreneurs. The founders of RYZ have worked at some of the world's largest tech companies and some of the most iconic consumer brands. They have lived and worked in Argentina for many years and have decades of experience in Latam. What brought them together was their passion for the early phases of company creation and the idea of attracting the brightest talents in order to build industry-defining companies in a post-pandemic world.

Our teams are remote and distributed throughout the US and Latam. They use the latest cutting-edge cloud computing technologies to create scalable and resilient applications. We aim to provide diverse product solutions for different industries and plan to build a large number of startups in the upcoming years.

At RYZ, you will find yourself working with autonomy and efficiency, owning every step of your development. We provide an environment of opportunities, learning, growth, expansion, and challenging projects. You will deepen your experience while sharing and learning from a team of great professionals and specialists.

Our values and