Ryzlabs1mo ago

AI QA Analyst

ArgentinaRemoteFull Time - Contractmid

Data ScienceQA & TestingQa Analyst

2 views0 saves0 applied

Apply Now

Quick Summary

Key Responsibilities

-Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH).

Requirements Summary

-Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content. -Experience managing, cleaning, or labeling datasets for machine learning or NLP projects.

Technical Tools

Data ScienceQA & TestingQa Analyst

At Ryz Labs, we are looking for an AI QA Analyst to join one of our clients, an AI-first travel and hospitality company. In this role, you will act as the bridge between model development and user experience. Your primary goal is to ensure our generative models are accurate, safe, and contextually aware by implementing rigorous evaluation frameworks and data-driven insights.

Qualifications:

-Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content.

-Experience managing, cleaning, or labeling datasets for machine learning or NLP projects.

-Familiarity with the AI lifecycle, Prompt Engineering, and tools like Google Sheets/Excel (SQL or Python is a significant plus).

Key Responsibilities:

-Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH).

-Build and maintain high-quality "Golden Datasets" and benchmarks to stress-test the model across various domains and edge cases.

-Conduct pre-deployment testing and A/B model comparisons to identify performance regressions or improvements.

-Categorize model failures (hallucinations, logic errors, tone drift) to provide actionable feedback to the Engineering and Research teams.

-Help define and refine the rubric for "what a good response looks like" as the product evolves.

-Building LLM-as-a-judge system with golden prompts and automation for output scoring

-Regression monitoring: detecting quality degradation across prompt, model, and config changes

-Evaluating agent routing

-AI Safety & Guardrails checks

Stack: Braintrust / Langsmith

About RYZ Labs:

RYZ Labs is a startup studio founded in 2021 by two lifelong entrepreneurs. The founders of RYZ have worked at some of the world's largest tech companies and some of the most iconic consumer brands. They have lived and worked in Argentina for many years and have decades of experience in Latam. What brought them together was their passion for the early phases of company creation and the idea of attracting the brightest talents in order to build industry-defining companies in a post-pandemic world.

Our teams are remote and distributed throughout the US and Latam. They use the latest cutting-edge cloud computing technologies to create scalable and resilient applications. We aim to provide diverse product solutions for different industries and plan to build a large number of startups in the upcoming years.

At RYZ, you will find yourself working with autonomy and efficiency, owning every step of your development. We provide an environment of opportunities, learning, growth, expansion, and challenging projects. You will deepen your experience while sharing and learning from a team of great professionals and specialists.

Our values and what to expect:

- Customer First Mentality - Every decision we make should be made through the lens of the customer.

- Bias for Action - urgency is critical, expect that the timeline to get something done is accelerated.

- Ownership - Step up if you see an opportunity to help, even if it's not your core responsibility.

- Humility and Respect - Be willing to learn, be vulnerable, and treat everyone who interacts with RYZ with respect.

- Frugality - being frugal and cost-conscious helps us do more with less

- Deliver Impact - get things done most efficiently.

- Raise our Standards - always be looking to improve our processes, our team, and our expectations. The status quo is not good enough and never should be.