Build the Golden Set First: A RAG Evaluation Harness for PMs

Building a RAG evaluation harness means creating a labeled set of real questions, expected source chunks, and expected answers before you build dashboards or automated scoring. Without that golden set, every retrieval tweak, prompt change, or model swap is judged by vibes — someone skims five outputs and declares victory. The golden set is what turns "seems better" into a measurable claim.

Quick Answer: A RAG eval harness needs a golden dataset first — 50-150 real queries, each with expected source chunks and a reference answer. Everything else (automated scoring, regression gates, dashboards) is built on top of that labeled set, not before it.

Most teams build the harness backwards. They wire up an LLM-as-judge pipeline, add a retrieval-quality dashboard, maybe integrate RAGAS or TruLens, and only then ask "what are we actually comparing against?" That question has no good answer without labeled data, so the fancy tooling ends up scoring against nothing. This piece treats the golden set as the deliverable that unlocks everything else — how to source real queries, what to put in each row, how big it needs to be, and how to grow it as production reveals failures you didn't anticipate.

Why the golden set has to come before the harness

A RAG evaluation harness without a golden dataset can compute metrics, but it can't tell you if those metrics mean anything — there's no ground truth to compare against, only relative comparisons between two equally-unverified runs. The golden set is the fixed reference point that makes "better" a testable claim instead of an opinion.

Think of it the way you'd think about a regression test suite in software engineering. You don't write a test suite by first building elaborate assertion-reporting infrastructure and later figuring out what to assert. You write test cases — input, expected output — and the reporting is secondary. RAG evaluation is the same shape of problem, except the "expected output" is fuzzier: a reference answer plus a set of source chunks that should have been retrieved to produce it.

Three consequences follow from skipping this step:

1. You can't detect regressions. A chunking change that quietly breaks retrieval for a whole document type looks fine until a real user complains — because nothing was checking that document type systematically.
2. LLM-as-judge scoring has nothing to calibrate against. Judge models drift, disagree with themselves across runs, and need a human-labeled anchor set to sanity-check against, per findings from Stanford's HELM evaluation project and Anthropic's own guidance on using models as evaluators.
3. Stakeholder trust erodes. When someone asks "did this retrieval change actually help?" and the answer is "I looked at a few examples," that's not evidence — and repeated instances of that answer cost credibility with engineering and leadership.

This is one of four product decisions RAG forces on a PM early — deciding what "correct" means for your system is not optional, and the golden set is where that decision gets written down concretely, row by row.

Sourcing real queries instead of inventing them

The fastest way to build a useless eval set is to sit in a room and imagine what users might ask. Real production queries — even a modest sample — surface phrasing, edge cases, and failure patterns that no PM brainstorming session invents on its own.

Where to pull real queries from

Pull from as many of these sources as you have access to, prioritizing ones closest to actual usage:

• Support tickets and chat logs — the literal questions customers already asked a human, which is the closest proxy for what they'll ask your RAG system.
• Search query logs from any existing search or help-center product, even a keyword-based one that predates the RAG system.
• Sales and onboarding call transcripts, where prospects ask the questions your documentation is supposed to answer.
• Internal Slack/Teams channels where employees ask each other things your knowledge base should cover.
• Existing FAQ pages, treated as a floor, not a ceiling — they cover known questions, not the long tail.

Don't rely on a single source. Support tickets skew toward complaints and edge cases; search logs skew toward short, ambiguous phrasing; sales calls skew toward pre-purchase questions. A golden set built from only one source inherits that source's blind spots.

Stratify by query type, not just topic

A common mistake is stratifying the golden set purely by subject matter (billing, onboarding, API docs) while ignoring query shape. Real usage includes several distinct shapes, and each stresses a different part of the pipeline:

Each row type belongs in the golden set deliberately, not incidentally. If your production traffic is 60% factual lookups, 25% multi-hop, and 15% everything else, your golden set's proportions should roughly track that — otherwise your aggregate score optimizes for a distribution of queries nobody actually sends.

What goes in a golden set row

Every row in the golden set needs enough structure that a human — or an automated judge calibrated against humans — can score a candidate answer without guessing at what "correct" means. Four fields do most of the work: the query, expected source chunks, an expected answer, and tags for slicing results later.

The template row

Use a flat, spreadsheet-friendly structure. It doesn't need a database — a shared spreadsheet or a lightweight table is enough to start, and it stays inspectable by non-engineers on the team.

Two fields deserve extra care. expected_sources is what lets you evaluate retrieval and generation separately — a wrong answer built on the right chunks is a generation bug; a right-sounding answer built on the wrong chunks is a retrieval bug that will eventually produce a confidently wrong answer on a harder query. This separation connects directly to how you structure chunks in the first place — if chunk boundaries are wrong, "expected sources" becomes a moving target every time you re-chunk.

expected_answer should be written the way you'd want the system to actually answer — not a copy-paste of a document paragraph, but a natural, concise response a support rep would give. If the true answer is "it depends," write that dependency into the reference answer rather than picking a single canonical phrasing.

Writing expected answers without overfitting to exact wording

Don't expect character-for-character matches — that's brittle and will fail on a phrasing you'd actually be happy with. Two practical approaches keep this reasonable:

1. Write the expected answer as a checklist of required facts, not a single sentence, so scoring (human or LLM-judge) checks for presence of each fact rather than string similarity.
2. Flag answers that have legitimate multiple correct phrasings with a flexible_wording tag so your scoring rubric treats them differently than factual-lookup rows where wording is tighter.

Sizing the golden set: start small, stratify deliberately

A golden set of 50-150 well-chosen rows is enough to start catching real regressions; it doesn't need thousands of rows before it's useful. Size matters less than coverage — a smaller set that spans your real query distribution beats a larger one clustered around easy cases.

A sizing framework

Rather than picking an arbitrary number, size around coverage targets:

• Minimum viable set (~50 rows): enough to cover your top 5-8 query types across your top 3-5 document areas, roughly 8-10 rows per combination. This is enough to catch a chunking regression that breaks an entire document type.
• Working set (~100-150 rows): adds edge cases, adversarial queries, and underrepresented topics identified from the first month of production traffic.
• Mature set (~300+ rows): built over months from production failures (see the cadence below), covering enough long-tail cases that a new model or retrieval change gets a statistically meaningful pass/fail signal, not just a handful of anecdotes.

Research on evaluation set design — including guidance from OpenAI's eval framework and academic work on few-shot benchmark construction — consistently finds that a stratified sample of a few dozen to a couple hundred well-labeled cases outperforms a much larger but poorly-stratified set for detecting regressions, because signal-to-noise matters more than raw volume. A golden set is a diagnostic instrument, not a training corpus — it doesn't need scale, it needs representativeness.

Don't let the set skew toward what's easy to label

It's tempting to fill the golden set with clean factual-lookup questions because they're fast to write reference answers for. Resist that. The multi-hop, ambiguous, and out-of-scope rows are harder to write and more valuable — they're exactly the query types where retrieval and generation quietly fail in production while factual-lookup queries keep passing.

Scoring against the golden set

A golden set only pays off once you run candidate answers against it and score both retrieval and generation. Score them separately — retrieval precision/recall against expected_sources, and answer quality against expected_answer — so a failure tells you which half of the pipeline to fix.

A minimal scoring rubric

Run this after any meaningful change — a new embedding model, a chunking strategy revision, a prompt rewrite, a reranking step. Treat a drop in retrieval hit rate on previously-passing rows as a hard regression signal, the same way a broken unit test blocks a merge.

This is also where access boundaries matter: a retrieval hit that's technically correct but pulled from a document the querying user shouldn't see is a failure mode your golden set should include test rows for, tied to the same permission and access-control decisions RAG systems force on the underlying architecture.

Growing the set from production failures

The golden set isn't static — its most valuable growth comes from real production failures: a support escalation, a low-confidence retrieval flagged by monitoring, a user thumbs-down on an answer. Build a lightweight, recurring cadence for triaging these into new golden rows rather than letting them disappear into a ticket queue.

A monthly cadence that keeps the set current

1. Weekly: tag any production interaction that got escalated, flagged, or thumbs-downed. Don't write full golden rows yet — just tag and queue.
2. Monthly: triage the queue. For each flagged case, decide: is this a genuine new failure mode, a duplicate of an existing golden row's pattern, or noise (a one-off user error)?
3. Monthly: for genuine new failure modes, write the full golden row — query, expected sources, expected answer, tags — and mark it with a source: production-failure tag so you can track how much of the set's growth is coming from real usage versus synthetic authoring.
4. Quarterly: re-check the set's stratification against current production traffic. If a query type or document area has grown as a share of real usage, add rows proportionally.
5. Quarterly: retire or revise golden rows tied to documentation or features that no longer exist — a stale golden set that scores against deprecated content produces misleading pass rates.

This cadence matters more than any single scoring tool. A golden set that never grows becomes a snapshot of last quarter's failure modes while production drifts elsewhere — the classic gap between an understanding of customer jobs at launch and how those jobs actually evolve once real usage starts talking back.

Where Prodinja fits into this workflow

Once you have a labeled golden set, the harder ongoing work is applying it consistently — judging retrieval quality and answer quality against curated expected cases every time something changes, rather than only when someone remembers to. Prodinja's Evals critique layer is designed to walk you through exactly that: structuring a golden case, checking retrieval and answer quality against it, and surfacing where a change helped or hurt. It's a framework for applying the discipline this article describes, not a replacement for building the golden set itself — that labeling work is still yours to do, and it's worth doing well regardless of what tooling sits on top of it.

If you're earlier in the RAG build than this — still deciding on chunking, retrieval architecture, or access model — the complete guide to RAG and knowledge systems is a useful starting point before the eval harness becomes the priority. Evaluation is the layer that tells you whether those earlier decisions are actually working, not a substitute for making them well in the first place. And if your RAG system sits behind a broader product experience, revisiting the customer journey around where users hit search or ask questions will tell you which query types deserve golden-set priority first.

Key Takeaways

• Build the golden set before the harness — labeled queries with expected sources and answers are the reference point that makes any metric meaningful.
• Source real queries from support tickets, search logs, sales calls, and existing FAQs — never invent them from a brainstorming session alone.
• Stratify by query type (factual, multi-hop, ambiguous, comparative, out-of-scope, adversarial), not just topic, since each type stresses a different part of the pipeline.
• A template row needs five core fields: query, expected_sources, expected_answer, answer_type, and tags — enough structure for consistent scoring.
• Start with 50-150 rows, sized around coverage of your top query types and document areas, not an arbitrary large number.
• Score retrieval and generation separately so a failure points to which half of the pipeline actually broke.
• Grow the set on a recurring cadence — weekly tagging, monthly triage and authoring, quarterly restratification and cleanup — driven by real production failures.

Frequently Asked Questions

How big should a RAG eval golden dataset be to start?

Fifty to 150 well-stratified rows is enough to start catching regressions, as long as they cover your top query types (factual, multi-hop, ambiguous, out-of-scope) across your main document areas. Size matters less than representativeness of real production traffic.

What's the difference between a golden dataset and an LLM-as-judge eval?

A golden dataset is the labeled ground truth — real queries with expected sources and answers. An LLM-as-judge is a scoring method that can run against that dataset, but it needs the human-labeled golden set to calibrate its own reliability; without it, the judge has nothing verified to compare its scores against.

Should the golden set include adversarial or out-of-scope queries?

Yes — out-of-scope and adversarial rows test whether your system refuses correctly or gets manipulated, which factual-lookup queries never reveal. Skipping them leaves a real failure mode (bad refusals, prompt injection, hallucinated answers to unanswerable questions) completely untested.

How often should a RAG golden dataset be updated?

Monthly for adding rows from tagged production failures, quarterly for restratifying against current traffic and retiring stale rows tied to deprecated content. A golden set that never grows drifts away from what users are actually asking within a few months.

Can one golden dataset work for both retrieval and answer-quality evaluation?

Yes, if each row includes both expected_sources (for retrieval scoring) and expected_answer (for generation scoring), scored independently. Splitting the two scores is what lets you diagnose whether a failure is a retrieval problem or a generation problem rather than one blended "wrong" verdict.