Shipping Products That Sometimes Lie: Managing Hallucination Risk as a PM

Your LLM feature will confidently generate wrong answers, and no amount of prompt tuning eliminates that entirely. The AI PM's real job is not chasing zero hallucinations — it's bounding the blast radius so a wrong answer never reaches a user in a context where being wrong is expensive.

Quick Answer: Treat hallucination as a risk to manage, not a bug to fix. Tier your AI features by the real-world cost of a wrong answer, then match mitigation intensity — grounding, citations, confidence signals, human review — to that tier.

Why "Fix the Hallucination" Is the Wrong Framing

Hallucination is a structural property of how large language models generate text, not a defect that a patch removes. LLMs predict statistically plausible next tokens; they have no built-in mechanism to distinguish a fact they retrieved from a fact they invented. Researchers at OpenAI, Anthropic, and academic labs studying factuality (including work summarized in Stanford's HAI reports) have consistently found that even state-of-the-art models hallucinate on some meaningful share of open-ended factual queries — the rate varies wildly by domain and prompt, but it never hits zero with current architectures.

That means a PM who greenlights a feature only after "hallucinations are fixed" will wait forever. The better question, borrowed from classic risk management, is: what is the worst plausible outcome if this specific output is wrong, and can we survive it?

This reframes your job in three ways:

1. You stop treating every hallucination as equally bad. A wrong product description and a wrong dosage recommendation are not the same incident.
2. You start designing mitigations proportional to consequence, not uniformly applying the same guardrail everywhere.
3. You accept residual risk explicitly, the way a security team accepts residual risk after mitigations, rather than pretending it's zero.

This is also where the discipline overlaps heavily with the broader shift in what the AI PM role actually covers versus traditional software PM work — trust and safety judgment becomes a core skill, not a side concern for legal to handle.

The Risk-Tiering Framework: Sort by Consequence, Not by Feature

The single highest-leverage thing a PM can do is build a risk tier for every generative surface before shipping it, because the tier — not the feature name — determines how much mitigation is warranted. Skipping this step is how low-stakes features get over-engineered and high-stakes ones get under-protected.

A workable tiering scheme has three levels, defined by what happens when the model is confidently wrong:

Two things make this table work in practice. First, tier by output, not by product. A single app can contain all three tiers — a legal-tech tool's "summarize this clause" feature might be medium-stakes while its "draft a binding contract term" feature is high-stakes. Second, re-tier when the audience changes. The same output aimed at an internal analyst versus an end consumer with no way to verify it can sit in different tiers.

How to Run the Tiering Exercise

Do this before writing a single line of prompt engineering, ideally in the same session where you'd normally scope a PRD:

• List every distinct generative output the feature can produce, not just the feature as a whole.
• Ask "what does the user do with this output?" — read it, act on it immediately, forward it to someone else, or use it to make an irreversible decision.
• Estimate reversibility. Can a wrong answer be caught and corrected before real-world harm, or does it propagate (a diagnosis acted on, a contract signed)?
• Assign a tier and write it down somewhere durable — a spec, a risk register, a hallucination-tier field in your tracker — so it survives past the person who did the exercise.

Grounding and Retrieval: Shrinking the Space Where the Model Can Invent

Grounding constrains an LLM to generate from a bounded set of retrieved, verifiable source material instead of its parametric memory, which directly shrinks the surface area where invention can happen. It doesn't eliminate hallucination — a model can still misread a retrieved passage — but it converts "the model made something up from nothing" into "the model misused a source you can audit."

Retrieval-augmented generation (RAG) is the dominant pattern: fetch relevant documents from a trusted corpus, inject them into the prompt as context, and instruct the model to answer only from that context. The quality of grounding depends on three levers, each of which is a real product decision, not just an engineering detail:

1. Corpus quality and freshness — retrieving from stale, low-quality, or unvetted documents just moves the hallucination upstream into your retrieval layer.
2. Retrieval precision — a retriever that pulls the wrong passages hands the model bad raw material even if the model itself behaves perfectly.
3. Instruction discipline — explicit prompting to refuse or flag when retrieved context doesn't answer the question, rather than falling back on parametric knowledge.

When RAG Isn't Enough

Grounding helps most for factual lookup tasks — "what does our refund policy say," "what did this contract clause specify." It helps less for generative or synthetic tasks where there's no single ground-truth document to retrieve — "write a persuasive summary," "suggest a next step." For those, confidence display and human review (below) carry more of the weight than retrieval does.

Citations, Confidence Display, and Human Review: The User-Facing Layer

Even well-grounded systems need user-facing signals, because grounding reduces the rate of hallucination but does nothing to help a user detect the ones that still happen. Three mechanisms do most of the work, and they compose — a mature high-stakes feature typically uses all three together.

Citations attribute each claim to its retrieved source, letting a user (or a downstream reviewer) verify the answer against the original document instead of trusting the model's paraphrase. Design citations to be specific — link to the passage, not just the document — because a citation to a 40-page PDF is barely more useful than no citation.

Confidence display surfaces the model's (or a secondary classifier's) uncertainty, but only works if it's calibrated against actual accuracy rather than being a cosmetic percentage. A poorly calibrated confidence score is arguably worse than none, because it manufactures false trust. If your team can't yet validate that "80% confidence" corresponds to roughly 80% empirical accuracy, don't ship a numeric score — use coarser signals instead, like:

• A qualitative label ("verify this" vs. "well-supported")
• Visual distinction between grounded and ungrounded portions of a response
• An explicit "I'm not certain" refusal path when retrieval confidence is low

Human review is the backstop for anything in the high-stakes tier, and the design question is where the human sits in the loop, not whether they're there at all:

Choosing the wrong pattern for a tier is a common failure mode: teams under launch pressure quietly slide a high-stakes feature from human-in-the-loop to human-after-the-loop because review is slow, which is exactly the shortcut the tiering exercise exists to prevent.

Working With the Team That Builds the Guardrails

None of the mechanisms above are a PM's to build alone — they require close, ongoing collaboration with the data scientists and ML engineers who tune retrieval, calibrate confidence scores, and instrument the audit trail. The relationship looks less like a hand-off spec and more like a shared experiment, which is the core distinction covered in how PMs work with data scientists on experiments versus features — hallucination mitigation is closer to an ongoing experiment than a one-time feature build.

That collaboration should also be structured around evaluation, not just launch. An eval-driven roadmap — where quality metrics like groundedness rate, citation accuracy, and refusal-appropriateness are tracked release over release — keeps hallucination mitigation from becoming a one-time launch checklist that quietly decays as the model, corpus, or user base shifts. This is the same discipline described in planning roadmaps around quality with evals, applied specifically to trust and safety metrics rather than general feature quality.

Metrics Worth Tracking Per Tier

• Groundedness rate — the share of claims in a response traceable to a retrieved source.
• Citation precision — whether cited sources actually support the claim attached to them.
• Escalation rate — how often human-in-the-loop review catches and blocks a bad output before it ships.
• User-reported correction rate — a leading indicator that your confidence display is miscalibrated if it's rising.

Where Hallucination Risk Diverges From a Traditional Software PM's Risk Model

Traditional software risk is largely deterministic: a bug reproduces the same way every time given the same inputs, so once it's fixed, it stays fixed. Hallucination risk is probabilistic and drifts with model updates, prompt changes, and shifting user inputs, which means a mitigation validated last quarter can silently degrade this quarter without any code change on your side. This is one of the sharper points of departure covered in the key differences between AI PM and traditional software PM work — a "closed" bug in this world is really a monitored, currently-acceptable risk level.

Practically, this means your trust-and-safety work never fully ships. Build in recurring review cadence — monthly for high-stakes tiers, quarterly for medium — where you re-check groundedness and escalation metrics against your original tier assumptions, because the model provider updating their base model underneath you is entirely outside your release cycle.

It also changes how you scope discovery work. Understanding what a user is actually trying to accomplish — the underlying job to be done — matters more here than in typical feature work, because the acceptable hallucination tolerance often depends less on the feature and more on what job the user has hired it for. A user drafting a first pass of marketing copy has enormous tolerance for a wrong turn of phrase; a user relying on the same underlying model to summarize a contract clause before signing has almost none, even though the interface might look identical.

Bringing It Together in the Product Lifecycle

Hallucination risk isn't a launch-week checklist item — it recurs at every stage where the user's context or stakes shift, which is easiest to see by mapping it onto the user's customer journey rather than treating it as a single point-in-time review.

1. Discovery — identify which generative surfaces exist and assign initial risk tiers.
2. Design — choose grounding architecture, citation format, and confidence display proportional to tier.
3. Build — instrument groundedness and citation-precision metrics from day one, not retrofitted after an incident.
4. Launch — set the human-review pattern (in-the-loop, on-the-loop, after-the-loop) matching the tier, and document the decision.
5. Post-launch — run recurring eval reviews; re-tier if the user base, model version, or use case shifts.

Skipping any one of these stages is usually where a real incident traces back to — not a single catastrophic model failure, but a tier that was assigned once at discovery and never revisited as the feature's actual usage evolved past its original scope.

Prodinja and Hallucination Risk

Key Takeaways

• Hallucination is a structural property of LLMs, not a bug — the PM's job is bounding consequence, not chasing zero.
• Tier every generative output by the real-world cost of a wrong answer, from low-stakes draft copy to high-stakes medical or legal outputs, and match mitigation intensity to the tier.
• Grounding and retrieval shrink where a model can invent answers but don't eliminate misuse of retrieved sources — corpus quality, retrieval precision, and instruction discipline all matter.
• Citations, calibrated confidence display, and the right human-review pattern (in-the-loop, on-the-loop, or after-the-loop) form the user-facing safety layer.
• Hallucination risk drifts over time with model and prompt changes, so mitigation needs a recurring review cadence, not a one-time launch checklist.
• This work is collaborative and cross-functional — close partnership with data science on evals and metrics is as important as any single guardrail you ship.