If a RAG system gives a bad answer, most teams debug the wrong layer. Retrieval metrics — recall@k, precision@k, MRR, and nDCG — measure whether your system fetched the right document chunks before generation ever ran. Without them, you're evaluating whether the answer sounded good, not whether it was built on the right evidence.
Quick Answer: Recall@k asks "was the right chunk anywhere in the top k results?" Precision@k asks "how much of the top k was actually relevant?" MRR asks "how quickly did we hit the first right answer?" nDCG asks "did we rank the best chunks highest, not just include them?" Measure all four before you touch generation.
Why Retrieval Metrics Are a Different Layer From Generation Metrics
Retrieval and generation fail independently, and conflating them is the single most common evaluation mistake in RAG products. Retrieval metrics measure whether the right source material was fetched from your vector index. Generation metrics measure whether the model used that material well — coherently, faithfully, without hallucinating beyond it.
A system can score high on a generation eval (fluent, well-formatted, confident) while retrieval quietly fed it the wrong chunk entirely. The model then does its job perfectly on bad inputs — and produces a wrong answer that reads as trustworthy. This is worse than an obviously broken output, because nobody flags it for review.
The Diagnostic Question That Separates the Two Layers
Ask this before any other debugging step: "Was the correct information present in the retrieved context at all?" If no, you have a retrieval problem — no amount of prompt engineering or better generation models fixes it. If yes but the answer is still wrong, you have a generation problem — the model ignored, misread, or under-weighted the right chunk.
Skipping this split leads teams to spend weeks tuning prompts and temperature settings against a retrieval bug. The complete guide to RAG for product managers covers where this fits into the broader system architecture; this article stays narrowly focused on the retrieval measurement layer itself.
- Retrieval failure looks like: confident answers built on irrelevant or outdated chunks.
- Generation failure looks like: the right chunk was retrieved, but the model summarized it wrong, missed a caveat, or hallucinated an addition.
- Both can fail simultaneously, and a generation-only eval can't tell you which one to fix first.
Recall@k: Did We Find the Right Chunk At All?
Recall@k measures whether at least one truly relevant document appears anywhere within the top k retrieved results, expressed as the fraction of queries where that's true. It's a binary hit-or-miss check per query, then averaged across your test set. It answers the most basic question: is the right information reachable at all, regardless of where it lands in the ranking.
Recall@k is intentionally forgiving about ranking position — it only cares that the relevant chunk made the cut. This makes it the right first metric to check, because a system with low recall@k has a retrieval ceiling problem: no re-ranking or prompt tuning downstream can recover information that was never fetched.
How to Calculate It
For a single query, recall@k = 1 if any relevant document is in the top k, else 0. Across a test set of n queries, recall@k = (queries with a hit) / n.
- Build a labeled test set of queries with known relevant document(s) per query.
- Run retrieval, capture the top k chunk IDs returned.
- Check whether any labeled-relevant chunk appears in that list.
- Average the hit/miss across all queries.
A recall@5 of 0.62 means 62% of test queries had their correct chunk somewhere in the top 5 results — and 38% of the time, the answer was structurally impossible to generate correctly, no matter how good your LLM is. This is often a symptom of a chunking decision made upstream; see how chunking strategy drives retrieval quality for the mechanics of why chunk boundaries directly cap recall.
Precision@k: How Much of What We Fetched Was Actually Useful
Precision@k measures the fraction of the top k retrieved chunks that are actually relevant, independent of whether all relevant chunks were found. It's the mirror image of recall: recall asks "did we miss anything," precision asks "did we drown the good stuff in noise."
High recall with low precision is a specific, recognizable failure pattern: the system finds the right chunk but buries it among four irrelevant ones. This matters because most generation models have a limited effective context window and can be distracted by irrelevant retrieved text — a phenomenon sometimes called the "lost in the middle" problem, documented in retrieval-augmented generation research from groups including Stanford's NLP lab. More retrieved chunks isn't free; each irrelevant one is a chance for the model to get pulled off track.
| Metric | Question it answers | Blind spot |
|---|---|---|
| Recall@k | Was the right chunk in the top k at all? | Ignores ranking position and noise |
| Precision@k | How much of the top k was relevant? | Ignores whether we missed a relevant chunk entirely |
| MRR | How fast did we hit the first correct chunk? | Ignores relevant chunks beyond the first hit |
| nDCG | Are the best chunks ranked highest, weighted by graded relevance? | Requires graded (not just binary) relevance labels to be worth using |
The table above is the core mental model to keep at hand — no single metric tells the whole story, which is exactly why teams that report only one (usually recall) get surprised later by a precision or ranking problem the single number never surfaced.
MRR: How Quickly Did We Surface the First Right Answer
Mean Reciprocal Rank (MRR) scores how early the first relevant result appears in the ranked list, averaged across queries, where earlier is exponentially better than later. For a single query, reciprocal rank is 1 divided by the position of the first relevant hit — rank 1 scores 1.0, rank 2 scores 0.5, rank 4 scores 0.25, and a miss scores 0.
MRR matters most in single-answer retrieval contexts — a chatbot that only feeds the model the top 1-3 chunks, or a search UI where users rarely scroll past the first result. It's less useful when your system deliberately retrieves a wide net (say, top 10) and depends on the generation layer to synthesize across several relevant chunks, because MRR only rewards the position of the first hit and ignores everything after it.
MRR's Practical Ceiling
A system that always ranks the correct chunk second, consistently, scores a flat 0.5 MRR forever — useful information, because it tells you the retriever is close but has a systematic ranking bias worth investigating (often an embedding model or re-ranker tuning issue), not a recall problem.
nDCG: Are the Best Results Ranked Highest, Not Just Present?
Normalized Discounted Cumulative Gain (nDCG) measures whether the most relevant results are ranked at the top, using graded relevance scores (not just relevant/irrelevant) and discounting the value of relevant results that appear lower in the list. It's the most complete of the four metrics because it accounts for degrees of relevance and rewards good ranking order, not just presence.
nDCG requires more setup than the other three: you need graded relevance labels (say, 0-3, where 3 is "directly answers the query" and 1 is "tangentially related") rather than simple binary relevant/irrelevant tags. This labeling effort is exactly why many teams skip nDCG and default to recall@k alone — but for any system where result ordering matters to the user or downstream model, that shortcut costs you visibility into a real failure mode.
The nDCG Formula, in Plain Terms
- DCG (Discounted Cumulative Gain): sum each result's relevance score, divided by the log of its rank position — so a highly relevant result at rank 5 contributes less than the same result would at rank 1.
- IDCG (Ideal DCG): the DCG you'd get if results were perfectly ordered by relevance, best first.
- nDCG = DCG / IDCG, normalized to a 0-1 scale so it's comparable across queries with different numbers of relevant documents.
An nDCG of 1.0 means your ranking is the best possible ordering available in your index for that query. An nDCG of 0.7 means relevant chunks exist and were mostly retrieved, but the ordering left value on the table — the ideal chunk might be sitting at rank 3 instead of rank 1.
Worked Example: Five Documents, One Query
Concrete numbers make these formulas click faster than definitions alone. Assume a query where your labeled ground truth says documents D2 and D4 are relevant (D2 highly relevant, graded 3; D4 partially relevant, graded 1), and your retriever returns this ranked order: D3, D2, D1, D4, D5.
| Rank | Document | Relevance grade | Relevant (binary)? |
|---|---|---|---|
| 1 | D3 | 0 | No |
| 2 | D2 | 3 | Yes |
| 3 | D1 | 0 | No |
| 4 | D4 | 1 | Yes |
| 5 | D5 | 0 | No |
Working through each metric against this single ranked list:
- Recall@5 = 1.0 — both relevant documents (D2, D4) appear somewhere in the top 5.
- Recall@2 = 0.5 — only D2 is captured in the top 2; D4 is missed at this cutoff.
- Precision@5 = 2/5 = 0.4 — two of five retrieved documents were relevant.
- Precision@2 = 1/2 = 0.5 — one of the top two documents was relevant.
- MRR = 1/2 = 0.5 — the first relevant document (D2) appears at rank 2.
- nDCG@5: DCG sums relevance/log2(rank+1) across positions — D2's grade-3 relevance at rank 2 dominates the score, while D4's grade-1 relevance at rank 4 contributes little; IDCG assumes the ideal order (D2 then D4, ranks 1-2), so nDCG comes out well below 1.0 here, quantifying exactly how much the ranking cost you by not surfacing D2 first.
Run this same worked calculation across your own test set and averaged, this is precisely the report a rigorous retrieval eval should produce — one row per metric, not a single vague "retrieval seems fine" impression from spot-checking three examples.
Building a Retrieval Test Set That Actually Works
A retrieval eval is only as good as its labeled ground truth, and building that set is where most teams underinvest relative to how often they'll lean on it. You need real queries — pulled from actual usage logs, not invented by the team building the retriever — each paired with a human judgment of which documents are genuinely relevant.
- Sample real queries from production logs or a beta user set, covering common patterns and known edge cases.
- Have a human (or a small panel) label relevance per query against your document set — binary for recall/precision/MRR, graded 0-3 for nDCG.
- Keep the set at 50-100+ queries minimum before trusting aggregate numbers; single-digit test sets swing wildly on one hard query.
- Refresh it periodically as your document corpus and query patterns evolve — a stale test set measures a retriever against a world that no longer exists.
This test-set discipline mirrors how the best UX research grounds itself in real behavior rather than assumption — the same instinct behind mapping actual customer jobs to be done rather than guessing at what users want done. Retrieval evaluation and JTBD research share a root principle: measure against reality, not intuition.
Where Prodinja Fits: Judging Retrieved Context, Not Just Answer Quality
Most eval tooling stops at "did the final answer sound right," which is exactly the blind spot this article has been walking through. Prodinja's Evals critique layer is designed to walk you through judging whether the retrieved context was actually the right context for a given query — not just whether the generated answer read well — surfacing the retrieval-versus-generation split as an explicit step in the review, rather than something a PM has to remember to check manually.
That framing matters because it's the same discipline this article argues for: separate the two layers before you draw conclusions about either one. If your team is deciding whether RAG or fine-tuning is the right approach for a given problem, having retrieval-quality visibility first changes that decision — a fine-tuning fix can't repair a recall problem any more than a better prompt can.
Key Takeaways
- Recall@k and precision@k answer different questions — recall checks whether the right chunk was found anywhere in the top k; precision checks how much of the top k was noise.
- MRR rewards speed to the first correct hit, making it most relevant for single-answer or top-3 retrieval contexts, not wide-net retrieval strategies.
- nDCG is the most complete metric because it uses graded relevance and penalizes good chunks buried at low ranks, but it requires more labeling effort upfront.
- Diagnose the layer before you diagnose the bug — ask "was the right information in the retrieved context?" before touching prompts, temperature, or the generation model.
- A labeled test set of real queries, refreshed periodically, is the foundation every one of these metrics depends on — invented queries produce misleading numbers.
- Chunking strategy is often the true upstream cause of low recall@k, so a retrieval metrics problem frequently traces back to a chunking decision, not the retriever itself.
Frequently Asked Questions
What's a good recall@k score for a RAG system?
There's no universal target — it depends on your document corpus size, query difficulty, and how many chunks (k) you feed the generator. As a practical calibration point, many production teams treat recall@5 or recall@10 below roughly 0.7-0.8 as a signal worth investigating, since it means the correct chunk is structurally missing from the generator's input in a meaningful share of queries.
Should I optimize for recall or precision in RAG retrieval?
Start with recall, because a precision problem is fixable downstream (re-ranking, filtering, tighter prompts) while a recall problem means the answer was never reachable. Once recall clears an acceptable threshold, shift attention to precision — otherwise you're stuffing the context window with noise the model has to filter itself.
How is nDCG different from MRR?
MRR only scores the position of the first relevant result and ignores everything after it; nDCG scores the whole ranked list using graded relevance, rewarding systems that put the most relevant results first, not just a relevant result first. Use MRR for single-answer contexts and nDCG when ranking quality across multiple results matters.
Do I need graded relevance labels, or is binary relevant/irrelevant enough?
Binary labels are sufficient for recall@k, precision@k, and MRR, and are far cheaper to collect. Graded labels (e.g., 0-3 scale) are required specifically for nDCG, because its value comes from distinguishing "somewhat relevant" from "exactly what the query needed" — if you only have binary labels, nDCG collapses to a less informative variant.
Can generation-quality metrics substitute for retrieval metrics?
No — they measure different failure modes and neither substitutes for the other. A generation eval judges whether the model used its given context well; it can't tell you whether that context was the right context to begin with, which is precisely the gap retrieval metrics are built to close.