A RAG answer is wrong for exactly one of two reasons: the system retrieved the wrong context, or it generated poorly from the right context. You can't tell which without instrumenting both stages separately—retrieval hit rate and chunk relevance on one side, context precision and answer faithfulness on the other.

Quick answer: Instrument retrieval and generation as two independent stages with their own metrics. If retrieved chunks don't contain the answer, it's a retrieval failure—fix indexing, chunking, or embeddings. If they do contain it and the model still answers wrong, it's a generation failure—fix the prompt or the model.

Why "The RAG Answer Was Wrong" Is a Useless Bug Report

A wrong answer alone tells you nothing actionable. It could mean the retriever pulled irrelevant chunks, the right chunk got buried below the top-k cutoff, or the generator had good context and still hallucinated. Each has a different fix, and they live in different code.

Most teams debug RAG the way they debug a black box: read the final answer, guess, and re-prompt. That's slow and it teaches you nothing about the system. The fix is to log and score each stage independently, every time, in production—not just in offline evals.

This is the same discipline behind closing the loop from failures to eval cases: a failure without a stage attribution is a failure you can't turn into a regression test. You need to know what to add to your eval set, not just that something broke.

The Two-Suspect Model

Think of every RAG failure as a police lineup with exactly two suspects:

  1. Retrieval failure — the context passed to the LLM didn't contain the information needed to answer correctly.
  2. Generation failure — the context was sufficient, but the LLM didn't use it correctly (ignored it, misread it, or hallucinated past it).

A third, rarer case exists too: both stages fail simultaneously, or the failure is upstream of both (bad chunking at ingestion time). We'll get to that. But start with the binary split—it resolves the majority of production incidents fast.

Retrieval Metrics: Did the System Find the Right Thing?

Retrieval quality monitoring answers one question: given the query, did the vector search (or hybrid search) surface chunks that actually contain the answer, ranked high enough for the generator to see them within its context window.

Retrieval Hit Rate

Retrieval hit rate (also called recall@k) measures whether the ground-truth-relevant chunk appears anywhere in the top-k results returned by the retriever. It's binary per query, averaged across a test set.

  • How to compute it: For a labeled eval set of (query, known-relevant-chunk-id) pairs, check if that chunk-id is in the top-k retrieved set.
  • Why it matters first: If hit rate is low, nothing downstream can save you. The generator literally never sees the right information.
  • Typical failure modes: stale embeddings after a document update, chunk boundaries that split an answer across two chunks, or a query phrased very differently from the source document's vocabulary (a classic vocabulary-mismatch problem that Robertson and Spärck Jones's original BM25 work already flagged decades before dense retrieval existed).

Chunk Relevance (Precision, Not Just Recall)

Hit rate tells you the right chunk exists somewhere in top-k. Chunk relevance, usually expressed as context precision, tells you how much of what you retrieved is actually useful versus noise.

MetricWhat it measuresFailure it catches
Retrieval hit rate (recall@k)Is the relevant chunk anywhere in top-k?Missing context entirely
Context precisionWhat fraction of retrieved chunks are relevant?Signal drowned in irrelevant chunks
Mean Reciprocal Rank (MRR)How high does the first relevant chunk rank?Relevant chunk buried too low
Chunk relevance score (LLM-judged)Does this specific chunk address the query?Per-chunk noise identification

A retriever can have a perfect hit rate and still hurt you if it returns eight chunks and only one is relevant. Long, noisy context windows measurably degrade generation quality—the "lost in the middle" effect documented by Liu et al. (Stanford, 2023), where models systematically under-attend to information buried in the middle of a long context. High recall with low precision doesn't just waste tokens; it actively increases the odds of a generation-stage failure that isn't the generator's fault.

Generation Metrics: Did the Model Use What It Was Given?

Once you know the context was adequate, the question shifts entirely: did the model produce an answer that's actually supported by that context, or did it wander off and hallucinate anyway.

Answer Faithfulness (a.k.a. Groundedness)

Answer faithfulness measures whether every claim in the generated answer can be traced back to the retrieved context. This is distinct from correctness against some external ground truth—faithfulness is purely about whether the model stayed inside the evidence it was handed.

  • How teams typically score it: an LLM-as-judge pass that checks each sentence of the answer against the retrieved chunks and flags unsupported claims, or a natural-language-inference (NLI) style entailment check.
  • What a low score reveals: the model is hallucinating or blending in outside "knowledge" instead of grounding in context—even when the context was good.
  • What a low score does not mean: it doesn't mean the answer is factually wrong in the real world. A faithful answer can still be wrong if the source document itself was wrong. Faithfulness and correctness are separate axes.

Answer Relevance

Separately, check whether the generated answer actually addresses the user's question, independent of faithfulness. A model can produce a perfectly faithful summary of the retrieved chunk that simply doesn't answer what was asked—common when retrieval returns adjacent-but-not-quite-right content and generation dutifully summarizes it anyway.

The Attribution Matrix: Assigning Blame to the Right Stage

Here's the decision logic, and it's simpler than most teams make it. Cross-reference retrieval quality against generation faithfulness for the failed query, and the stage announces itself.

Retrieval qualityGeneration faithfulnessDiagnosisFix
Low (chunk missing/irrelevant)N/ARetrieval failureFix chunking, embeddings, or indexing
High (right chunk present)Low (answer contradicts context)Generation failureFix prompt, add citations, try a different model
HighHigh, but answer still wrongSource data or ground-truth failureCheck the underlying document, not the pipeline
LowLowCompound failureFix retrieval first, re-test generation after

Run this matrix on every flagged production failure, not just eval-set failures. This is the connective tissue that turns quality drift with no stack trace into quality drift with a very clear stack trace: a stage-by-stage score trail attached to every response.

The Case of Perfect Generation, Wasted on Garbage

Consider a support-docs RAG bot asked: "What's the retry policy for failed webhook deliveries?" The retriever, tuned on semantic similarity alone, pulls three chunks: one about webhook signing, one about rate limits, and one about general error codes. None mention retry policy.

The generator, given zero relevant context, does the responsible thing: it says it doesn't have enough information to answer confidently, or worse, it politely paraphrases the rate-limits chunk as if it were about retries. Judged on faithfulness alone, that paraphrase might even score reasonably—it's grounded in a chunk, just the wrong one.

This is the trap: a faithfulness score can look fine while the real defect is 100% upstream. If your dashboard only tracks generation-stage metrics, this failure is invisible—the model behaved exactly as instructed given garbage input. The retrieval hit rate for this query is 0%, and that's the number that should have paged someone. This is precisely why three dashboards on day one should include a retrieval-stage view distinct from your generation-quality view—collapsing them into one "RAG quality" number hides exactly this kind of failure.

Instrumenting the Pipeline: What to Log at Each Stage

You can't score what you don't capture. At minimum, log these artifacts per request, not aggregated—per-request traces are what let you replay a specific failure later.

  • Query as sent to the retriever (post any query rewriting/expansion)
  • All retrieved chunk IDs, scores, and rank positions, not just the top one
  • The exact context string assembled for the prompt (after truncation, deduplication, reordering)
  • The generated answer verbatim
  • Any citations or chunk references the model produced, if your prompt requests them

This is the same "observe everything, judge selectively" posture covered in the broader LLMOps observability guide—the logging has to be unconditional; the scoring can be sampled or triggered by user feedback.

Sampling Strategy for Production Judging

Running an LLM-judge faithfulness check on every single production request is expensive and often unnecessary. A reasonable approach:

  1. Log 100% of requests (cheap—just structured data capture).
  2. Score 100% of retrieval-stage metrics (hit rate proxies, chunk count, score distribution—all cheap, deterministic, no LLM call needed).
  3. Sample generation-stage LLM-judge scoring on a rolling percentage, plus 100% of anything flagged by a user thumbs-down or support escalation.
  4. Route every flagged failure through the attribution matrix before it goes into a bug ticket.

Where Retrieval Problems Actually Start

Most retrieval failures don't originate in the vector database or the embedding model—they originate upstream, in how the knowledge base's entities and relationships were structured in the first place. If your source data conflates "customer" and "account" as one loosely-typed blob, no amount of embedding-model tuning fixes queries that need one but retrieve the other.

This is a modeling problem before it's a retrieval problem, which is exactly the reasoning Prodinja's Data Modelling studio is built around: walking through the entities and relationships behind a knowledge base—turning them into structured schema (down to SQL DDL)—before you ever touch chunking strategy. A clean entity model doesn't guarantee good retrieval, but a muddled one guarantees confused chunks, and confused chunks show up as retrieval-stage failures no matter how good your embedding model is.

Key Takeaways

  • Split RAG failures into two suspects immediately: retrieval (wrong/missing context) and generation (misused context). Never debug the combined output as one blob.
  • Track retrieval hit rate and context precision separately—high recall with low precision still degrades generation via context dilution and "lost in the middle" effects.
  • Answer faithfulness measures groundedness, not correctness—a faithful answer can still be wrong if the source document was wrong, so don't conflate the two.
  • Run every flagged failure through an attribution matrix cross-referencing retrieval quality against generation faithfulness before filing a bug.
  • Perfect generation on irrelevant chunks is a retrieval failure wearing a generation costume—watch for dashboards that only surface generation-stage metrics.
  • Log all pipeline artifacts unconditionally, but sample the expensive LLM-judge scoring—retrieval metrics are cheap enough to run on everything.
  • Retrieval problems often start upstream in data modeling, not in the vector index—muddled entities produce muddled chunks regardless of embedding quality.

Frequently Asked Questions

How do you know if a RAG failure is retrieval or generation?

Check whether the retrieved chunks actually contain the information needed to answer the query. If they don't, it's a retrieval failure; if they do but the answer contradicts or ignores them, it's a generation failure—this is what the attribution matrix formalizes.

What is context precision in RAG observability?

Context precision measures what fraction of the chunks a retriever returns are actually relevant to the query, distinct from recall (whether the relevant chunk exists in the results at all). Low precision means real signal is diluted by noise, which can degrade generation even when the right chunk is technically present.

What's the difference between answer faithfulness and answer correctness?

Answer faithfulness checks whether every claim in the answer is supported by the retrieved context, regardless of external truth. Answer correctness checks the answer against ground truth. A generated answer can be faithful to a wrong source document and still be factually incorrect.

Why does a RAG pipeline need separate dashboards for retrieval and generation?

Because a single blended "RAG quality" score can look acceptable while masking a 0% retrieval hit rate on a specific query category—the generator can score well on faithfulness by faithfully summarizing irrelevant chunks, hiding the real defect from anyone watching only generation metrics.

How often should you run LLM-as-judge faithfulness scoring in production?

Log every request's retrieval and generation artifacts unconditionally, since that capture is cheap, but sample LLM-judge faithfulness scoring on a rolling percentage plus 100% of user-flagged or escalated failures, since judge calls carry real cost and latency.