A user's literal last message is often the worst possible string to send to a retriever — it's missing the context, the vocabulary, and the specificity that retrieval needs. Query rewriting transforms that raw message into one or more better-formed queries before the search happens, using techniques like decontextualization, acronym expansion, multi-query generation, and HyDE.
Quick Answer: Rewrite the user's raw message before retrieval, not after. Decontextualize follow-ups, expand jargon, generate several phrasings, and consider HyDE for conceptual questions — each adds latency and cost, so match the technique to the query's difficulty.
Why the literal last message fails
The most common retrieval failure isn't a bad embedding model or a broken index — it's feeding the retriever a message that was never designed to stand alone. Conversational input is elliptical by nature; humans drop context they assume is shared, and a retriever has no memory of the conversation to fill the gap.
Consider a real support-bot exchange:
- User: "Does the enterprise plan support SSO?"
- Bot: "Yes, SAML and OIDC are both supported on Enterprise."
- User: "What about the one below that?"
Send message 3 literally to a vector index and you get near-nothing useful. There's no entity in it — "the one below that" has zero lexical or semantic overlap with "Business plan SSO support," which is presumably what's actually being asked. The embedding model can't infer a plan hierarchy it never saw stated in the same string.
This is the gap query rewriting exists to close: turning a query that makes sense to a person mid-conversation into one that makes sense to a search index with no memory. Our complete guide to RAG frames retrieval quality as a chain of compounding decisions — rewriting is the first link, and a weak one undermines everything downstream, no matter how good your chunking strategy is.
The under-specification problem, beyond follow-ups
Even standalone first messages are frequently under-specified. "How do I fix the auth error" doesn't say which auth provider, which SDK version, or which error code — a retriever matching on those literal words will surface generic auth documentation instead of the one page that actually resolves the user's specific error.
Vocabulary mismatch compounds this. Users write in their own words; documentation is written in the source's words. A user asking about "the thing that stops me overpaying twice" is describing idempotency keys without knowing the term — pure lexical or dense-vector similarity on the raw query may never bridge that gap without an explicit rewriting step in between.
Decontextualizing follow-ups
Decontextualization rewrites a context-dependent follow-up into a self-contained query that carries the necessary history explicitly, so retrieval doesn't need conversational memory to succeed. Done well, "What about the one below that?" becomes "Does the Business plan support SSO?" — a string a retriever can actually match against.
The standard pattern uses an LLM call, seeded with recent turns, whose only job is producing a rewritten query:
"Given the conversation history and the user's last message, rewrite the last message as a standalone question containing all necessary context. Output only the rewritten question."
This works because decontextualization is a narrow, well-bounded task — much narrower than open-ended generation — which makes it a good candidate for a smaller, cheaper model than the one answering the user. A few practical notes:
- Resolve pronouns and deictic references ("it," "that," "the one below," "this feature") to their named referents from prior turns.
- Carry forward implicit filters — if turn one established "for the enterprise plan," a later "is it SOC 2 compliant" should inherit that scope.
- Preserve negation and comparison carefully; "not the enterprise one, the tier below" is a common failure point where naive rewriting drops the "not."
- Stop rewriting once the query is already self-contained — running every message through a rewrite step adds latency for no benefit on turn-one questions.
A worked multi-turn example
| Turn | Raw message | Rewritten query sent to retriever |
|---|---|---|
| 1 | "Does the enterprise plan support SSO?" | Does the enterprise plan support SSO? (unchanged — already standalone) |
| 2 | "What about the one below that?" | Does the Business plan support SSO? |
| 3 | "And is that included or an add-on?" | Is SSO included in the Business plan, or is it a paid add-on? |
Notice turn 3 depends on turn 2's rewritten output, not the raw message — decontextualization should chain against its own prior outputs, not just the raw transcript, or errors compound turn over turn.
Expanding acronyms, jargon, and implicit entities
Acronym and entity expansion widens a query's surface area so it matches how the same concept is phrased in the knowledge base, catching the mismatch between how users write and how documents are written. A user typing "CAC" might mean customer acquisition cost, and if your corpus spells it out fully in prose, a literal-string retriever misses the connection entirely.
This matters more in domain-specific corpora than general ones. Internal wikis, legal documents, and technical specs are dense with organization-specific shorthand a general-purpose embedding model was never trained to disambiguate. A retrieval layer that expands SLA to "service level agreement," or a product codename to its public name, closes a gap no amount of chunking optimization fixes.
Practical approaches, roughly in order of effort:
- A maintained glossary/dictionary mapping acronyms and internal codenames to expansions, injected as a rewrite step — cheapest, most deterministic, but requires upkeep as vocabulary shifts.
- LLM-based expansion, prompting the model to append plausible full forms alongside the original acronym, useful when the domain vocabulary is too large or fast-moving to maintain by hand.
- Query-side synonym injection at the search-engine layer (common in traditional BM25/lexical setups), which pairs well with a hybrid retrieval approach that blends lexical and dense signals.
Expansion is a case where a small, boring dictionary often outperforms a clever model — if your domain's acronym list is finite and stable, hardcode it.
Generating multiple query variants
Multi-query generation asks an LLM to produce several differently-phrased versions of the same underlying question, retrieves for each in parallel, and merges the results — trading one retrieval call for several to reduce the odds that a single unlucky phrasing misses the right document. A user's exact wording is one lens into their intent; a handful of paraphrases widens the net without requiring a smarter embedding model.
For "How do I stop getting double-charged for renewals," useful variants might include:
- "Prevent duplicate billing on subscription renewal"
- "Idempotency for recurring payment charges"
- "Avoid double invoices on auto-renew"
Each variant is likely to have different overlap with different chunks in the index — merging their top-k results (commonly via reciprocal rank fusion or simple deduplication-and-union) surfaces documents that any single phrasing alone would have missed.
When multi-query earns its cost
| Scenario | Multi-query worth it? | Why |
|---|---|---|
| High-stakes support answer with no fallback | Yes | Missing the right chunk has a real cost; parallel variants reduce that risk |
| High-traffic FAQ-style queries with known good phrasing | Usually no | The literal query already retrieves reliably; added latency buys little |
| Long-tail, ambiguous, or exploratory questions | Yes | Single phrasing is unlikely to match the corpus's actual wording |
| Latency-sensitive real-time chat (sub-second budget) | Case-by-case | Parallel calls can be fired concurrently, but each still adds a model round-trip |
The core trade is recall versus latency and cost — more variants generally raise the odds of hitting the right chunk, but each variant is another embedding call (and often another LLM call to generate it), so this is a deliberate budget decision, not a free upgrade.
HyDE: retrieving with a hypothetical answer
HyDE (Hypothetical Document Embeddings), introduced by researchers at Carnegie Mellon University and the University of Waterloo, asks an LLM to write a plausible answer to the query first, then embeds that hypothetical answer — not the query itself — to search the index. The insight is that a hypothetical answer's prose is often closer, in embedding space, to a real document's prose than the original question's phrasing ever was.
Take "What causes thermal throttling in laptops?" A direct query embedding captures the question shape; a hypothetical answer like "Thermal throttling occurs when a CPU's temperature sensor detects it has exceeded a safe threshold, triggering a reduction in clock speed to prevent damage" reads structurally like the documentation it's trying to find — same declarative, explanatory register.
HyDE tends to help most on conceptual, explanatory queries where the gap between question-phrasing and answer-phrasing is largest, and least on queries that are already lookups (a product name, an exact error code), where the literal query already matches well.
Watch two failure modes:
- Hallucinated specifics. If the LLM's hypothetical answer invents a plausible-sounding but wrong fact (a made-up API parameter, a fabricated statistic), the embedding search chases that fabrication instead of the real content — HyDE's quality is bounded by the generating model's domain knowledge.
- Added latency stacking with other techniques. Combining HyDE with multi-query multiplies generation calls before a single retrieval happens, so it's rarely the default — it's a technique to reach for on a specific query class, not a blanket policy.
Weighing the latency and cost trade-off
Every rewriting technique inserts at least one additional model call before retrieval starts, which is a real, compounding cost that a PM should size deliberately rather than default into. A single decontextualization call might add 200-500ms and a fraction of a cent; stacking decontextualization, three-way multi-query, and HyDE on every message can turn a sub-second retrieval into a multi-second wait with several times the per-query cost.
| Technique | Typical added latency | Typical added cost | Best-fit query type |
|---|---|---|---|
| Decontextualization | One small-model call (~100-400ms) | Low | Multi-turn follow-ups |
| Acronym/entity expansion | Near-zero if dictionary-based; one call if LLM-based | Very low to low | Domain-heavy, jargon-dense corpora |
| Multi-query (3-5 variants) | One generation call + N parallel retrievals | Moderate (N embedding calls) | Ambiguous, long-tail, high-stakes queries |
| HyDE | One generation call before embedding | Low to moderate | Conceptual, explanatory questions |
The pragmatic pattern most teams converge on is tiered, not uniform: classify incoming queries cheaply first (is this a follow-up? does it contain a known acronym? is it conceptual or a lookup?), and route only the queries that need it to the heavier techniques. Applying HyDE to "What's your refund policy?" is wasted latency; applying nothing to "What about the one below that?" is a guaranteed miss.
This routing decision sits squarely inside what our piece on the four product decisions RAG forces calls out — retrieval architecture isn't a single setting, it's a set of trade-offs a PM has to make explicitly, including whether the answer belongs in a RAG pipeline at all versus fine-tuning for stable, narrow domains where rewriting overhead may not be worth carrying at all.
Measuring whether rewriting is actually helping
Don't ship a rewriting layer on faith — instrument it. Track retrieval recall on a held-out query set (does the right chunk appear in top-k) with and without each rewriting technique, and pair it with latency percentiles, not just averages, since a rewriting call's tail latency is what users actually feel. If a technique doesn't move recall meaningfully on your real query distribution, cut it — added latency without added recall is pure cost.
Where this becomes a product decision, not just an engineering one
Query rewriting isn't a backend implementation detail a PM can hand off and forget — it's a set of trade-offs about latency, cost, and which query types get special handling, and those are product calls. Deciding whether follow-ups get decontextualized, which acronyms are worth a maintained glossary, and whether HyDE runs on every conceptual question or none of them shapes the user's actual experience of "does this thing find what I asked for."
That same instinct — resisting the raw, literal signal in favor of one reshaped with context — shows up elsewhere in product work too. Interpreting what a customer actually needs from what they literally said is the whole premise behind Jobs to Be Done, and behind customer journey mapping: the stated request and the underlying intent are rarely identical, and the gap between them is exactly where the useful work happens.
Key Takeaways
- The literal last message is frequently the wrong retrieval query — conversational ellipsis, vocabulary mismatch, and under-specification all break naive retrieval before it starts.
- Decontextualization rewrites context-dependent follow-ups into standalone queries by resolving pronouns and carrying forward implicit scope from prior turns.
- Acronym and entity expansion closes the gap between how users write and how documents are written, especially in jargon-dense internal corpora.
- Multi-query generation trades one retrieval call for several paraphrased ones, raising recall on ambiguous or long-tail questions at a real latency and cost premium.
- HyDE embeds a hypothetical answer instead of the raw question, helping most on conceptual queries where question-phrasing and answer-phrasing diverge — but it inherits the generating model's hallucination risk.
- None of these techniques are free — tier them by query type rather than applying every technique to every message, and measure recall lift against latency cost before committing.
- Rewriting is a product decision, not just plumbing — deciding which techniques apply to which queries shapes the user's actual experience of retrieval quality.
Frequently Asked Questions
What is query rewriting in RAG?
Query rewriting in RAG is the step where a user's raw input is transformed — through decontextualization, expansion, or reformulation — into one or more queries better suited to retrieval, run before the vector or lexical search happens. It exists because raw conversational input is often ambiguous, under-specified, or missing context a retriever has no way to infer on its own.
Does query rewriting always improve retrieval results?
Not always — it improves results on queries suffering from the specific problem a given technique targets, and adds pure latency and cost on queries that don't. A well-formed, standalone, jargon-free query gains little from decontextualization or expansion; applying it anyway wastes a model call for no recall improvement.
What's the difference between query expansion and multi-query generation?
Query expansion adds terms (synonyms, spelled-out acronyms) to a single query to widen its matching surface, while multi-query generation produces several separate, differently-phrased queries and retrieves for each independently before merging results. Expansion enriches one search; multi-query runs several searches and combines them.
Is HyDE better than simple query rewriting?
Neither is universally better — HyDE tends to outperform simple rewriting on conceptual, explanatory questions where question-phrasing diverges sharply from answer-phrasing, while simple decontextualization or expansion is cheaper and sufficient for follow-ups and jargon-heavy lookups. Many production systems combine both, routed by query type, rather than picking one exclusively.
How much latency does query rewriting add to a RAG pipeline?
A single rewriting call (decontextualization or HyDE) typically adds one small-model round trip, often in the low hundreds of milliseconds, while multi-query generation adds that plus N parallel retrieval calls. The exact figure depends heavily on model choice and infrastructure, which is why measuring your own tail latency, not assuming a industry-wide number, matters before shipping any rewriting layer.