Retrieval works best when it's specced as one callable tool an agent can invoke on its own judgment, not a mandatory pre-step wired in front of every prompt. The agent decides whether the question needs a search, then judges what comes back — by source authority and freshness — and follows an explicit rule for what to do when nothing matches instead of quietly generating an answer anyway.

Quick Answer: Don't hardcode "always retrieve first." Spec retrieval as a tool with a trigger condition, a source-ranking rule, and a defined empty-result behavior (usually: cite or escalate, never guess).

Why "retrieval-as-pre-step" breaks down in practice

Most early RAG implementations bolt a vector search onto the front of every request: embed the query, fetch the top-k chunks, stuff them into the context window, generate. It works for a demo. It breaks the moment a real user asks something the retrieval corpus doesn't cover, because the pipeline still runs, still returns something, and the model still generates confidently from thin or irrelevant context.

The failure isn't the retriever — it's the architecture that never lets the agent decide not to retrieve, or decide retrieval failed. A fixed pipeline treats search as plumbing. An agentic design treats search as a decision the agent makes and can revisit.

This distinction matters more as agents take on multi-turn, tool-using work rather than single-shot Q&A. Our guide to agentic workflows covers the broader shift from linear pipelines to agents that choose their own tool sequence — retrieval is simply the first tool most knowledge agents need to get right.

The pre-step pattern and where it fails

In a classic RAG pipeline, retrieval always fires, regardless of query type. Ask "what's our refund policy for enterprise customers in the EU" and it searches. Ask "summarize what I just told you" and it still searches, wastes a retrieval call, and sometimes pollutes the context with irrelevant chunks that degrade the answer.

  • Chit-chat and meta-questions don't need a document lookup at all.
  • Ambiguous queries need clarification before search, not search on a bad query.
  • Multi-hop questions need two or three retrieval calls in sequence, not one.
  • Out-of-scope questions need the agent to recognize the corpus can't help — not a forced answer from the nearest, unrelated chunk.

A fixed pre-step handles none of these gracefully because it never asks "should I search, and for what?"

Treat retrieval as a callable tool with a trigger condition

An agent should decide to call search_knowledge_base the same way it decides to call any other tool: because the current state of the task requires information it doesn't already have. This means writing an explicit trigger condition into the tool's spec, not leaving it to implicit prompt-level hope.

A good trigger condition names the signal, not just the intent. "Search when the user asks a factual question" is too vague for a model to act on consistently. "Search when the answer requires a specific policy number, date, dollar figure, or named clause the agent hasn't already retrieved in this conversation" is something a model can actually check against.

Writing the trigger clause

Borrow structure from the five-part agent spec: goal, inputs, tools, guardrails, and success criteria. Retrieval's trigger condition belongs in the tools section, but it should reference the goal directly.

  1. Name the query classes that require retrieval — factual claims, policy citations, anything with a version or date attached.
  2. Name the query classes that don't — clarifications, summaries of prior turns, general reasoning the agent can do from its own training.
  3. Set a re-retrieval rule — if the conversation moves to a new sub-topic, does the agent search again, or reuse the first result set?
  4. Cap the number of calls per turn — an ungated agent can loop on search-then-reformulate indefinitely; two or three attempts is a common ceiling before escalation.

This is also where the goal-writing discipline from writing an agent goal without drift pays off — a goal phrased as "answer using retrieved sources" invites the agent to retrieve for everything; a goal phrased as "answer accurately, using retrieval only when the answer requires a specific fact" gives the model room to skip the tool when it should.

Source ranking: not every retrieved chunk deserves equal trust

Once retrieval fires, the agent needs a way to judge which returned passages to actually use — vector similarity alone tells you a chunk is topically close, not that it's authoritative, current, or even correct. Spec a ranking policy that layers authority and freshness signals on top of raw similarity score, and make the agent apply it explicitly rather than trusting whatever the vector store returns first.

Robert Cialdini's research on authority as a persuasion signal has a useful, literal analog here: readers (and models) should weight a claim differently depending on who's making it. A retrieval agent needs the same discipline encoded as policy, not left to the model's unguided judgment of which snippet "sounds right."

Ranking factorWhat it signalsHow to encode it
Source tierOfficial policy doc vs. Slack thread vs. old wiki pageMetadata tag at ingestion; hard floor below which a chunk is excluded
RecencyWhether the passage reflects current rulesTimestamp field; decay function or hard cutoff for time-sensitive domains
Similarity scoreTopical closeness to the query embeddingStandard vector distance; necessary but not sufficient alone
CorroborationWhether multiple independent sources agreeCount distinct source documents; flag single-source claims for lower confidence
Author/ownerWhether the source has an accountable ownerMetadata; deprioritize anonymous or unowned documents

A practical policy combines at least two of these — similarity plus source tier, at minimum — instead of similarity alone. Don't let recency silently override authority: a stale official policy is usually still more trustworthy than a fresh but unofficial forum post, unless the domain is explicitly time-sensitive (pricing, security advisories, live incident status).

Freshness deserves its own rule, separate from ranking

Freshness isn't just one more ranking input — in some domains it's a hard gate. A retrieval agent answering questions about pricing, compliance deadlines, or security advisories needs a cutoff, not a preference. Spec it as: "if the most relevant source is older than N days for this document category, treat it as unverified and say so" rather than folding age into a single blended score where a stale-but-strong match can still win.

  • Static domains (historical facts, established methodology) can weight recency lightly or not at all.
  • Volatile domains (pricing, policy, compliance, security) need an explicit staleness threshold that triggers a lower-confidence response or an escalation, not silent use of an outdated chunk.
  • Mixed domains need per-category thresholds — a single global freshness rule usually under- or over-corrects for at least one category.

The empty-result problem: where hallucination actually starts

Most hallucination in retrieval-augmented agents doesn't come from the model inventing facts out of nowhere — it comes from the agent treating a weak or empty retrieval result as if it were a strong one, and generating a fluent answer anyway because nothing in the architecture stops it. This is the single highest-leverage failure mode to spec against explicitly.

The fix is a named, mandatory branch in the agent's logic: define what "no source matches" means numerically (a similarity threshold, a minimum source-tier requirement, or both), and give the agent an explicit instruction for that branch that is not "answer from general knowledge."

Designing the escalation path

For a policy agent — one that must cite a specific document for every substantive claim — the empty-result branch should never fall through to free generation. It should do one of a small set of defined things instead.

  1. State the gap plainly: "I couldn't find a policy document that addresses this specific question."
  2. Offer the closest adjacent match with a confidence caveat, if one exists above a lower secondary threshold, clearly labeled as approximate.
  3. Escalate to a human owner — the stakeholder or team responsible for that policy area — rather than guessing on their behalf.
  4. Log the gap as a signal that the knowledge base itself has a hole, which is a data point worth acting on independently of the single conversation.

Anthropic's own guidance on tool use recommends giving models explicit permission and instruction for "I don't have enough information" outcomes rather than assuming a well-crafted prompt alone will suppress confident-sounding fabrication — the instruction has to be as concrete as the happy-path instruction. Guardrail language belongs in the same tier of spec as the tool's trigger condition, not left as an afterthought at the end of a prompt.

Escalate, don't guess: an agent that says "I don't know, and here's who to ask" is doing its job. An agent that fills the gap with a fluent, unsupported answer is the actual product risk.

Tying escalation to least-privilege design

The escalation path also intersects with tool scoping. An agent that can only read from an approved, tiered document set — and nothing broader — is structurally less likely to retrieve an untrustworthy source in the first place. Our piece on least-privilege agent tool access covers scoping retrieval tools down to exactly the corpora a given agent should be allowed to search, which reduces how often the empty-result and low-authority-source problems even arise.

Worked example: a policy agent that cites or escalates

Consider an internal HR policy agent fielding "can I expense a conference registration if I'm not presenting?" The trigger condition fires — this is a factual policy question with a specific answer, not small talk. Retrieval runs against the HR policy corpus, tagged and tiered at ingestion.

ScenarioRetrieval resultAgent behavior
Strong matchOfficial expense policy doc, updated this quarter, above similarity floorAnswer directly, cite the specific clause and document
Weak matchOld team wiki page, no official source foundState the caveat, cite the weak source explicitly as unofficial, offer to escalate
No matchNothing above the minimum thresholdDo not answer from general knowledge; state the gap; route to HR policy owner
Conflicting matchesTwo official docs disagree (e.g., old vs. new policy version)Surface both, flag the conflict, escalate for a human decision rather than picking one silently

The fourth row is easy to miss in a first draft of the spec — conflicting authoritative sources are a distinct failure mode from no source, and deserve their own branch rather than falling into either the strong-match or no-match path.

Grounding this in real user need, not just architecture

Before speccing any of this, it's worth validating that "cite a source or escalate" is actually the job the user needs done, versus a faster, looser answer they'd prefer even with lower certainty. The jobs to be done framework is useful here: an HR policy lookup is a job with real compliance stakes, where the customer's underlying "hire" for the agent is confidence and coverage, not speed — which argues strongly for escalation over guessing. A different agent, answering low-stakes internal FAQ questions, might reasonably tolerate a looser, faster-but-less-certain path. Map the customer journey around the moment a user hits an empty result, too — that's often the point where trust in the whole agent is won or lost, not a peripheral edge case.

How Prodinja frames this in practice

Prodinja's Agentic Workflows tool is designed to walk you through specifying retrieval the same way this article argues for — as one tool among several, not an assumed pre-step. When you define a retrieval tool in the prototype's guided flow, it prompts you to write out the trigger condition that decides when the agent searches, the ranking or freshness rule it applies to what comes back, and the explicit failure mode for a thin or empty result set. It's a structured way to force the three decisions this piece covers onto the page before the agent spec is considered done, rather than discovering the gaps once the agent is already answering questions with weak sources.

Key Takeaways

  • Spec retrieval as a callable tool with an explicit trigger condition, not a fixed pre-step that runs on every query regardless of need.
  • Name the query classes that require search versus the ones that don't, so the agent isn't retrieving for chit-chat or reusing stale context for a new sub-topic.
  • Rank sources on more than similarity — layer in source tier, recency, corroboration, and ownership so a topically-close but low-authority chunk doesn't win by default.
  • Give freshness its own rule for volatile domains (pricing, compliance, security) rather than blending age into one similarity score.
  • Define the empty-result branch explicitly: state the gap, offer a caveated adjacent match, or escalate to a human owner — never fall through to unsupported generation.
  • Conflicting authoritative sources are a distinct case from no source at all, and deserve their own escalation path in the spec.
  • Scope the retrieval tool's document access narrowly to reduce how often low-authority or irrelevant sources are retrievable in the first place.

Frequently Asked Questions

What's the difference between RAG and an agent that uses retrieval as a tool?

Classic RAG runs retrieval as a fixed pipeline step before every generation call, regardless of whether the query needs it. An agent using retrieval as a tool decides, per turn, whether to call search at all — based on an explicit trigger condition — and can skip it, call it multiple times, or escalate instead.

How do you rank retrieved sources beyond vector similarity?

Layer at least one authority signal — a source-tier tag applied at ingestion — on top of raw similarity score, and add recency where the domain is time-sensitive. A practical minimum is similarity plus source tier; corroboration across multiple independent documents is a useful third signal for high-stakes answers.

What should an agent do when retrieval returns nothing relevant?

It should not fall back to answering from general model knowledge. Spec an explicit empty-result branch: state that no matching source was found, optionally offer a caveated weaker match, and escalate to a human owner for anything with real stakes attached, such as policy or compliance questions.

How many times should an agent retry a search before giving up?

There's no universal number, but capping retries at two or three attempts per turn is a common ceiling before the agent should escalate rather than keep reformulating the query indefinitely. Without an explicit cap, an ungated agent can loop on search reformulation and burn latency and cost without a better result.

Does retrieval freshness matter for every type of question?

No — static domains like historical facts or established methodology can weight recency lightly, while volatile domains like pricing, policy, or security advisories need an explicit staleness threshold. Treating freshness as a single global rule usually under-corrects for volatile categories or over-corrects for stable ones.