When Your Sources Disagree: Designing for Conflicting Context

When two retrieved documents disagree, a large language model doesn't detect a conflict — it silently picks one and presents the result as fact. Reliable systems don't leave that to chance; they encode an explicit precedence rule, by recency, authority, or deliberate surfacing, so "which source wins" becomes a stated design decision instead of whatever the retriever happened to rank first.

Quick answer: Give every retrieved source a machine-readable priority signal — a timestamp, an authority tier, or an explicit rank — and resolve conflicts with that rule before generation, not by hoping the model adjudicates correctly. When two sources are genuinely tied, surface the conflict to the user instead of guessing.

Why Two Correct-Looking Sources Can Break an LLM Answer

Retrieval ranks documents by semantic similarity to the query, not by truth or currency, so a stale pricing page and this quarter's update can both score high enough to land in the same context window together. The model has no built-in way to know which one is authoritative, so it blends them, contradicts itself, or arbitrarily favors whichever chunk sits closest to the question.

This is a context problem, not a prompt problem. No amount of rewording the question fixes it, because the ambiguity lives in the retrieved material itself, not in how it's asked — a distinction worth internalizing if you're still treating prompt engineering and context engineering as interchangeable skills. You can craft the most precise instruction in the world and still hand the model two contradictory facts with no way to tell them apart.

Research on this is more concrete than intuition alone. In a 2023 study on knowledge conflicts in large language models, researchers found models don't resolve contradictory retrieved evidence consistently — they can swing toward whichever side of a conflict has more supporting passages, behaving less like a fact-checker and more like a vote-counter. A related 2024 benchmark testing how models handle context that contradicts their own correct internal knowledge found they sided with the retrieved — but wrong — passage in a majority of trials. That's the uncomfortable finding for anyone building retrieval-augmented systems: adding more context without a precedence rule doesn't make an answer more reliable. It can make a wrong answer more confidently stated.

The practical symptoms show up as instability, and they're recognizable once you know to look for them:

• The same question, asked twice, returns two different answers depending on retrieval order or chunk boundaries.
• Numbers get blended into a nonsensical average instead of either source being used cleanly.
• The model cites the wrong version of a document with full confidence, because nothing told it the version mattered.
• Answers silently favor whichever chunk landed closest to the question in the prompt — a position-in-context bias that has nothing to do with which source is actually more current, and is easy to confuse with genuine recency.

That last point matters enough to repeat: recency-in-the-prompt (where a chunk sits in the context window) and recency-of-the-source (when a document was actually published or updated) are different things, and only one of them should ever drive a resolution rule. Getting this straight is foundational enough that it's worth treating as part of your broader context engineering practice rather than a one-off fix.

Three Ways to Resolve a Source Conflict: Recency, Authority, or Surfacing

Every workable approach to conflicting context reduces to one of three moves: prefer the newest source, prefer the most trusted source regardless of age, or stop pretending there's one right answer and show the user both. Each trades reliability for a different kind of risk, and most mature systems end up combining at least two.

Recency wins is the default most teams reach for first, because it's cheap: attach a timestamp at ingestion and let the newest chunk take precedence whenever two chunks make conflicting claims about the same fact. It works well for anything with a clear "current version," like API references or pricing.

Authority wins borrows directly from master data management practice, where the discipline of maintaining one "golden record" per entity — the practice DAMA International's data management body of knowledge documents in detail — exists precisely because timestamp alone isn't a reliable signal of correctness. A vendor's official spec sheet, published once and rarely touched, should usually beat a frequently-edited internal wiki page even though the wiki looks "newer."

Surfacing the conflict is the move teams under-use, mostly because it feels like an admission of failure. It isn't. The U.S. National Institute of Standards and Technology's AI Risk Management Framework explicitly calls out provenance and traceability as core to trustworthy AI systems — telling a user "these two sources disagree, here's why" is a transparency practice, not a shortcoming.

It's worth being clear about what retrieval itself can and can't do here. The embedding model's job is to find chunks that are semantically similar to the query — it has no concept of which one is true. Choosing a stronger embedding model changes how precisely you retrieve relevant passages; it does nothing to adjudicate between two relevant passages that disagree. That's a separate design layer, which is why picking an embedding model and designing source precedence are two distinct decisions that both belong in a mature retrieval stack — solving one doesn't solve the other.

A Versioned-Docs Example: When the Old Integration Guide and the New Changelog Disagree

Picture a support assistant retrieving both the original integration guide and the current API changelog for one question: "how do I authenticate?" Without a precedence rule, the model may blend the two into an answer that recommends neither format correctly, or pick whichever chunk happened to embed closer to the query that particular run.

Here's what the retriever actually pulls back for that query:

Run each strategy from the table above against this exact set of chunks and you get three different behaviors:

1. Recency alone correctly promotes the changelog, since it's both the newest and the intended current behavior — a case where recency happens to line up with correctness.
2. Authority alone, if it ranks "official docs" as one flat tier without a recency tiebreaker, can't actually resolve anything — both the guide and the changelog are "official," so the conflict just moves one level up instead of being solved.
3. Surfacing catches something the other two miss entirely: the unverified forum thread. Neither recency nor authority alone would ever promote it, yet it flags a real edge case — silent failures on older clients — that the official docs don't mention at all.

A surfaced answer for this example might read:

Docs disagree here: the 2022 integration guide describes query-parameter auth, but the 2024 changelog marks that deprecated in favor of a header. Use the header method — it's current. Note: a community report suggests older v2 clients may fail silently with the header method; confirm your client version if you're integrating against an older SDK.

That answer takes longer to write and to read than a clean one-liner, but it's honest about what the sources actually say, including the part neither "official" source addresses. This exact failure pattern tends to recur at specific points along a user's path with your product — someone in an early research phase finds the outdated guide through a search engine while a newer email links the current changelog, and if support tooling can't tell which one trumps which, mapping the customer journey is often how you first notice where the stale document is still surfacing from at all.

Designing the Precedence Rule Into Your Context Packet

A precedence rule only works if it's attached to the data itself, not left to the model's judgment at inference time. Every chunk entering the context window needs machine-readable metadata — source, timestamp, authority tier — so a deterministic step decides what to keep, demote, or flag before generation starts, rather than hoping the model infers it from prose.

That metadata belongs inside the context packet the model actually receives, not in a separate lookup table the generation step never sees. If you haven't mapped out what a well-formed packet should carry, the anatomy of a context packet is worth reading alongside this, since precedence fields are exactly the kind of structural addition that's easy to bolt on badly and clean to design in from the start.

In practice, that means a short pipeline:

1. Tag at ingestion, not at retrieval. Attach a timestamp and an authority tier to every document the moment it enters your knowledge base, so the information is always available, not reconstructed on the fly.
2. Run a conflict-detection pass before generation. Flag chunks that make overlapping claims with different values — this is a cheap, deterministic check, not a job for the generation model itself.
3. Apply your precedence rule in code. Recency, authority, or a combined rule — resolved by a script, not by a prompt instruction hoping the model "prefers the newer one."
4. Pass the resolved hierarchy into the prompt explicitly. Something like "Primary source (Nov 2024): ...; Superseded source (Mar 2022), for reference only: ..." gives the model an unambiguous instruction instead of two equally-weighted facts.

None of this replaces solid retrieval underneath it. Precedence design assumes retrieval already found the collision in the first place — if your embedding choice doesn't cluster the old guide and the new changelog as the same underlying topic, the two chunks never meet in the same context window, and you won't know there's a conflict to resolve at all.

When to Surface the Conflict Instead of Resolving It

Automatic resolution is the right default for low-stakes, high-volume questions, but forcing a silent pick on a genuinely ambiguous or high-stakes conflict just trades a visible problem for an invisible one. The better default there is telling the user what disagrees and why, then letting them decide with the actual context of what they're trying to accomplish.

Which failure mode is worse — a wrong automatic answer, or a slower, caveated one — depends heavily on the job the person is actually hiring the answer to do. A developer debugging a production outage at 2 a.m. wants the fastest correct answer, full stop. A compliance officer checking a regulatory threshold wants to see the disagreement and decide for themselves. Thinking in terms of Jobs to Be Done is a useful way to decide, case by case, where automatic precedence is acceptable and where the extra friction of surfacing is worth paying.

A rough rule of thumb for where to draw that line:

• Auto-resolve when: the stakes of a wrong answer are low, the conflict is clearly about one source being stale rather than genuinely disputed, and volume is too high to put a human in the loop every time.
• Surface when: the numbers genuinely diverge on something consequential (financial, medical, legal, safety-related), the sources are tied on both recency and authority, or getting it wrong is expensive or hard to reverse.

Neither move is free. Auto-resolution risks a confidently wrong answer nobody catches until it causes damage. Surfacing risks a caveated answer that a busy reader skims past and misreads anyway. Choosing between them, deliberately, for each class of question your system handles, is the actual design work — not picking one strategy and applying it everywhere.

Key Takeaways

• Retrieval ranks by similarity, not by truth. Two contradictory sources can both score high enough to land in the same context window, and the model has no innate way to tell which one is right.
• This is a context-layer problem, not a prompt-layer one. Better wording doesn't resolve an ambiguity that lives in the retrieved material itself.
• Recency, authority, and surfacing are the three real strategies, and mature systems usually combine at least two rather than relying on one alone.
• Position-in-context and source-recency are different signals. Confusing "which chunk sits closest to the question" with "which document is actually newest" produces unstable, hard-to-debug answers.
• Precedence metadata has to travel with the source, tagged at ingestion and resolved in code before generation — not inferred by the model at answer time.
• Surfacing a conflict is a transparency practice, not a failure state, especially for high-stakes or genuinely tied sources where a silent pick is the riskier choice.
• The right default depends on the job the answer serves, not a single rule applied uniformly across every question a system handles.