A hallucination is a confidently-worded, fluent answer that is factually wrong or unsupported by any source the model actually had — and it is caught in production through three converging checks: groundedness scoring against retrieved context, self-consistency sampling across repeated generations, and citation verification that confirms a cited source actually says what's claimed.
Quick Answer: You don't detect hallucinations by looking for a "hallucination signature" in the text — there isn't one. You detect them by checking the answer against something external: the retrieved passages, multiple independent samples of the same prompt, or the literal content behind a citation. All three run as production monitors, not just pre-launch tests.
Hallucination Is a Distribution Problem, Not a Bug
A hallucination isn't a defect you patch with a regex or a stricter prompt — it's an expected output of a model sampling from a probability distribution over plausible-sounding text. When the distribution is well-grounded in retrieved context, sampled tokens track the source. When it isn't, the model still produces fluent, syntactically perfect text, just untethered from any document.
This distinction matters because it changes what "fixing" even means. A traditional software bug is deterministic — same input, same wrong output, every time, and a patch eliminates it permanently. A hallucination is probabilistic — the same prompt can produce a grounded answer nine times and a fabricated one on the tenth, with no code change in between. You can shift the odds (better retrieval, tighter prompts, a smaller allowed answer space) but you cannot patch the tail away.
Anthropic and OpenAI have both published guidance in this vein: model cards for GPT-4 and Claude describe hallucination rates as a property to be measured and reduced across releases, not a checkbox to close. Stanford's HELM benchmark project treats factuality as a continuous evaluation axis for exactly this reason — it moves.
Why "confident-but-wrong" is the hard case
The failure mode that actually damages trust isn't the model saying "I'm not sure." It's the model stating a fabricated detail with the same fluency and hedge-free tone as a correct one. Users — and most automated smell tests — cannot distinguish the two from the text alone.
- Surface fluency is not a factuality signal. Grammatical confidence and factual grounding are produced by different mechanisms and vary independently.
- Wrong answers cluster around plausible-sounding specifics — a real-looking API endpoint, a specific-sounding statute number, a citation to a paper that doesn't exist. Specificity increases perceived trust without increasing accuracy.
- The gap only shows up when someone checks. That's the whole problem: detection has to happen before a user is the one doing the checking.
Groundedness Checks: Scoring the Answer Against What Was Retrieved
A groundedness check scores whether each claim in a generated answer is actually supported by the passages the retrieval system handed the model — not whether the claim is true in some absolute sense, but whether this specific answer is traceable to this specific context. It's the single highest-leverage check for RAG systems because it isolates a distinct failure: the model ignoring context it was given.
The mechanic, in practice, has three common implementations:
- NLI-style entailment scoring. Run each generated sentence against each retrieved passage through a natural-language-inference model and check whether the passage entails, contradicts, or is neutral toward the claim. Neutral or contradicted spans get flagged.
- LLM-as-judge groundedness prompts. Ask a second model call (often a cheaper or more constrained one) "does this passage support this specific sentence, yes/no, cite the span." Frameworks like
RAGASandTruLensimplement variants of this pattern as a scoreable metric. - Span-attribution / highlighting. Some pipelines force the generator itself to emit which retrieved chunk backs each sentence, then verify the chunk actually contains that content — closer to citation verification (below) but run at the sentence level.
Groundedness only measures faithfulness to context, not correctness in the world. A model can be perfectly grounded in a retrieved document that is itself wrong, outdated, or off-topic. Groundedness and retrieval quality are separate axes and both need monitoring — see how this splits out in the complete guide to LLMOps observability.
The RAG example: retrieval was fine, the model ignored it
Consider a support bot answering "what's the refund window for annual plans?" The retriever correctly surfaces the exact policy paragraph stating a 30-day window for annual plans. The retrieval step, measured on its own precision/recall metrics, looks perfect — the right document, top of the ranked list, well within the context window.
The model, however, generates: "Annual plans have a 14-day refund window, consistent with our standard policy." It's fluent, on-topic, and completely wrong — it appears to have pattern-matched to a different, more common 14-day window it saw during pretraining or in an unrelated retrieved snippet, rather than reading the one document that mattered.
This is the case that makes teams overinvest in retrieval and underinvest in generation-side checks. A dashboard tracking retrieval precision alone would show green the entire time this defect was live — the failure is invisible to any metric that stops at "did we fetch the right document." Only a groundedness check comparing the generated claim against the retrieved text catches the mismatch, because it's the one signal placed after generation, not before it.
| Failure layer | What looks fine | What actually broke | Detection mechanism |
|---|---|---|---|
| Retrieval | Top-ranked document is correct and relevant | N/A — retrieval succeeded | Precision/recall, MRR |
| Generation | Answer reads fluently, on-topic, confident | Model ignored retrieved fact, substituted a memorized prior | Groundedness/entailment scoring |
| Citation | A source is named and looks plausible | Cited source doesn't contain the claim | Citation verification (below) |
Self-Consistency Sampling: Using Disagreement as a Signal
Self-consistency sampling generates the same prompt multiple times at nonzero temperature and measures how much the answers agree; low agreement on a factual claim is itself a hallucination signal, independent of whether any single answer looks confident. It works because a model's memorized, well-grounded knowledge tends to reproduce stably across samples, while a fabricated detail tends to vary.
The technique, sometimes called SelfCheckGPT in the research literature (Manakul et al., 2023), formalizes an intuition PMs already have about flaky humans: ask the same question five different ways and see if the answers hold together. In production it's typically run as:
- N-sample generation (commonly 3-10 samples) at moderate temperature for a subset of production traffic, not every request — cost makes 100% sampling impractical at scale.
- Pairwise or clustering agreement scoring — semantic similarity or NLI-based contradiction detection across the sample set, not exact string match.
- Variance-as-risk flagging — high-variance answers get routed to a stricter model, a human review queue, or an "I'm not certain" fallback rather than shipped as-is.
Where self-consistency is worth the cost — and where it isn't
Self-consistency sampling multiplies inference cost by the sample count, so it isn't something you run on every request. It earns its keep on high-stakes, low-volume factual queries — medical, legal, financial claims — where a single wrong answer costs more than the extra inference spend across a thousand cheaper queries.
It's poorly suited to open-ended creative or conversational outputs, where "disagreement" between samples is expected and not a defect signal at all. Match the technique to the query type, or you'll spend budget flagging normal variation as risk.
Citation Verification: Trusting Nothing the Model Attributes
Citation verification checks that a source the model names or quotes actually exists and actually contains the claim attributed to it — a distinct failure mode from ungrounded generation because the citation itself can be entirely fabricated even when surrounding text sounds sourced. Models will invent plausible-looking paper titles, page numbers, and URLs with the same fluency they use for real ones.
- Existence check: does the cited document, URL, or case citation resolve to something real? A dead link or a paper title that doesn't appear in any index is an immediate red flag.
- Content check: if it exists, does the passage actually say what's attributed to it? This is the more common failure — real source, misquoted or extrapolated claim.
- Specificity check: oddly precise numbers ("in a 2019 study of 4,217 participants") attached to a vague or absent citation are a known hallucination tell — models fabricate false precision more readily than they fabricate vague claims.
Retrieval-augmented systems reduce fabricated citations relative to ungrounded generation, but don't eliminate them — the model can still misattribute or misquote a real retrieved source. Citation verification has to run as its own check layer, not be assumed as a side effect of RAG.
Stanford's 2024 legal-AI hallucination study (Dahl et al.) is a useful reference point here: even retrieval-augmented legal research tools showed non-trivial rates of citations that were real cases but didn't support the stated proposition — the "content check" failure above, not the "existence check" one. That distinction is exactly why citation verification needs two separate passes, not one.
Building the Production Monitor, Not Just the Pre-Launch Test
The three checks above only earn their value running continuously on live traffic, sampling a percentage of production responses and scoring them asynchronously — not as a one-time pre-launch eval suite that never runs again. A model that passed your test set at launch will still drift as traffic mix, retrieval corpus, and prompt patterns change.
- Sample a percentage of production responses (often 1-10%, weighted toward high-stakes query types) for groundedness scoring, run asynchronously so it doesn't add latency to the user-facing path.
- Alert on rate changes, not absolute thresholds alone — a groundedness score dropping from a stable 92% to 78% over a week is a signal even if 78% sounds tolerable in isolation. This is the same "distribution shift" framing worth reading in why quality drift doesn't throw a stack trace.
- Route flagged responses to a queue, not directly back to the model — a human or a stricter secondary check should adjudicate before you decide whether it was a false positive or a real gap.
- Feed confirmed hallucinations back into eval cases so the same failure mode is caught pre-deploy next time, closing the loop between what production surfaced and what your test suite checks — the mechanics of that loop are covered in closing the loop from failures to eval cases.
- Track the metric on one of your core dashboards, not buried in a model-team-only tool — factuality belongs alongside latency and cost as a day-one signal, as laid out in the three dashboards every AI product needs on day one.
What this looks like for a PM prototyping the monitor
A factuality check you prototype in Prodinja's Evals concept — scoring a sample response against its retrieved context for groundedness — is designed to become the same online monitor pattern described above: the check that works on a handful of test cases pre-launch is the same shape of check that, running continuously, samples production traffic for ungrounded claims. It's a simulated critique experience in the prototype today, not a live scoring pipeline, but the design intent is that the eval case and the production monitor are the same artifact viewed at two points in its life, not two separate builds.
Choosing Detection Tactics by Failure Mode
Different hallucination failure modes call for different detection tactics — running all three everywhere is wasteful, but running only one leaves blind spots the others exist specifically to cover.
| Tactic | Catches | Misses | Cost profile |
|---|---|---|---|
| Groundedness scoring | Model ignoring or contradicting retrieved context | Wrong-but-grounded source content | Low — one extra scoring call per response |
| Self-consistency sampling | Unstable, low-confidence factual claims | Consistently-wrong claims the model always gets the same way | High — N× inference cost |
| Citation verification | Fabricated or misquoted sources | Ungrounded claims made without any citation at all | Medium — external lookup per citation |
Most production factuality monitors combine groundedness scoring as the always-on baseline with targeted self-consistency sampling on high-stakes query classes and citation verification wherever the product surfaces sourced claims at all.
Key Takeaways
- Hallucination is a probability-distribution problem, not a deterministic bug — the same prompt can be right nine times and wrong the tenth, so "fixing" it means shifting odds, not patching a single failure.
- Groundedness checks compare generated claims against retrieved context and are the highest-leverage detector for RAG systems because they catch the model ignoring good retrieval, not just bad retrieval.
- Retrieval-quality dashboards can look perfectly healthy while generation quietly fails — the RAG failure mode where the right document was fetched but the model substituted a memorized prior is invisible to retrieval-only metrics.
- Self-consistency sampling turns disagreement across repeated generations into a risk signal, best reserved for high-stakes, low-volume factual queries where the extra inference cost is worth it.
- Citation verification needs both an existence check and a content check — a real, resolvable source can still be misquoted or stretched beyond what it actually supports.
- Detection only works as a continuous production monitor, sampling live traffic and feeding confirmed failures back into your eval suite — not as a one-time pre-launch test that goes stale as usage drifts.
Frequently Asked Questions
How do you detect LLM hallucinations in production without checking every response manually?
Sample a percentage of live traffic (often 1-10%, weighted toward high-stakes queries) and run automated groundedness scoring asynchronously against the retrieved context, escalating only flagged responses to human review rather than reviewing every output by hand.
What's the difference between a hallucination and a retrieval failure?
A retrieval failure means the system fetched the wrong or no relevant document; a hallucination means the model generated an unsupported claim regardless of what was retrieved — including cases, as shown above, where retrieval was flawless and the model ignored it anyway.
Can self-consistency sampling replace groundedness checks?
No — they catch different failures. Self-consistency flags claims that vary across repeated samples, but a model can be consistently wrong the same way every time, which self-consistency alone won't catch; groundedness scoring against source context catches that case instead.
Does RAG eliminate hallucinations?
No — retrieval-augmented generation reduces ungrounded fabrication but doesn't eliminate it, since the model can still ignore correct retrieved context or misquote a real cited source, both of which require their own detection checks layered on top of RAG.
How much does hallucination detection cost to run at scale?
Groundedness scoring is relatively cheap (one extra scoring call per sampled response); self-consistency sampling multiplies inference cost by the number of samples generated, so it's typically reserved for a narrow slice of high-stakes queries rather than run on all traffic.