A reranker is a second-stage scorer that re-orders the candidates your first-stage retriever already pulled, trading extra latency for meaningfully better precision at the top of the list. It sits between retrieval and generation, reading each candidate against the query with far more attention than embeddings allow, then promotes the ones actually worth showing the model.
Quick Answer: First-stage retrieval (bi-encoder or BM25) casts a cheap, wide net over thousands of documents. A reranker (usually a cross-encoder) then scores only the top 20-100 candidates precisely and reorders them. Add one when retrieval recall is fine but precision at the top isn't — not as a default on every pipeline.
Most teams that shipped a RAG pipeline in the last two years reach for reranking only after users start noticing the model "ignoring" documents that were clearly retrieved. The documents were in context. They just weren't near the top, and the model — like a person skimming a stack of search results — pays most attention to whatever comes first. That's the problem reranking solves, and it's one of the few RAG upgrades that's genuinely close to a free lunch: no retraining, no new index, no schema change.
What a Reranker Actually Does in the Retrieve-Then-Rerank Pipeline
A reranker takes the candidates your retriever already found and re-scores each one against the query using a much more expensive, much more accurate model. It doesn't search the whole corpus — it only ever looks at the shortlist retrieval handed it, which is exactly what keeps the extra cost bounded.
The pattern has two distinct stages, and conflating them is the single most common design mistake:
- First-stage retrieval (recall-oriented): a bi-encoder or lexical method (
BM25,SPLADE) scans the full corpus — potentially millions of chunks — and returns the top-k(commonlyk=50to200) candidates fast, using precomputed vector similarity or an inverted index. - Second-stage reranking (precision-oriented): a cross-encoder reads the query and each candidate together, token by token, and outputs a single relevance score per pair. Only the top
kfrom stage one gets this treatment.
This two-stage shape isn't unique to RAG. It's the same retrieve-then-rank pattern that has run production web search and recommender systems for two decades — the retrieval funnel narrows a firehose into a shortlist before anything expensive gets applied to it. RAG pipelines are simply the newest system to rediscover it. If you're still deciding whether your architecture needs this level of retrieval sophistication at all versus a simpler approach, the RAG vs. fine-tuning decision is worth resolving first — reranking only matters once retrieval is the chosen path.
Why Retrieval Alone Leaves Precision on the Table
Bi-encoders retrieve fast because they encode the query and every document independently, into fixed vectors, ahead of time. Similarity becomes a single dot-product comparison — cheap enough to run against millions of vectors in milliseconds.
That independence is exactly what caps their precision. A bi-encoder never lets the query and the document "talk to each other" during encoding; it can't weigh which specific words in the document matter for this specific query. Two documents can land at nearly identical embedding distance from a query while differing sharply in actual relevance, and the bi-encoder has no mechanism to tell them apart. That gap is what a cross-encoder is built to close.
Bi-Encoders vs. Cross-Encoders: The Latency/Quality Tradeoff
Bi-encoders and cross-encoders trade the same resource in opposite directions: bi-encoders sacrifice precision for speed at corpus scale, while cross-encoders sacrifice speed for precision on a small candidate set. Neither is strictly better — they're built for different points in the pipeline.
A bi-encoder (the model behind most vector search, e.g. sentence-transformers-style embedding models) encodes query and document separately into vectors, then compares them with cosine similarity or dot product. Because document embeddings are precomputed and indexed once, query-time cost is just one new encoding plus a nearest-neighbor lookup — this is what makes sub-100ms search over millions of documents possible at all.
A cross-encoder concatenates the query and a candidate document into a single input and passes both through a transformer jointly, letting every token attend to every other token before producing one relevance score. That joint attention is more accurate precisely because it's expensive — there's no way to precompute it, since the score depends on the query and document pair together, not either one alone.
| Dimension | Bi-Encoder (retrieval) | Cross-Encoder (reranking) |
|---|---|---|
| Encoding | Query & document encoded separately | Query & document encoded jointly |
| Precomputable | Yes — document vectors indexed ahead of time | No — score depends on the specific pair |
| Scale | Millions of documents, sub-100ms | Dozens to low hundreds of candidates |
| Relevance signal | Coarse semantic similarity | Fine-grained, query-aware relevance |
| Typical latency per query | ~10-50ms for retrieval | ~50-300ms+ for reranking 50-100 candidates |
| Role in pipeline | Recall — don't miss the right document | Precision — surface it near the top |
The practical read: retrieval decides what's in the room; reranking decides who's called on first. Skipping the second stage means the model's attention gets spent in whatever order embedding math happened to produce, which is rarely the order that matters.
Where Latency Actually Goes
Reranking cost scales with candidates scored, not corpus size — so the lever you control is k, not the index. Reranking 100 candidates with a mid-sized cross-encoder typically adds tens to a few hundred milliseconds, depending on model size and batching; reranking 500 candidates can meaningfully change your latency budget.
- Narrow
kbefore reranking, not after — retrieval's job is to hand the reranker a shortlist worth the expense, not a firehose. - Batch cross-encoder calls rather than scoring candidates one at a time; most serving frameworks support this natively.
- Consider a lighter cross-encoder (a distilled variant) if latency, not accuracy, is your binding constraint.
The Decision Rule: When a Reranker Earns Its Cost
Add a reranker when your retrieval recall is already reasonable but precision at the top few positions is visibly weak — not as a default step bolted onto every pipeline regardless of whether it's needed. If the right document usually isn't even in your top-k candidates, reranking can't fix that; the fix is chunking strategy or retrieval tuning upstream, not a smarter scorer downstream.
Use this as a rough triage, not a formula:
| Signal | What it suggests |
|---|---|
| Right answer is in top-50 retrieved, rarely in top-5 | Classic reranking win — recall is fine, ranking isn't |
| Right answer is often missing from top-50 entirely | Fix retrieval/chunking first; reranking has nothing to promote |
| Query volume is low, latency budget is generous (batch/async use cases) | Reranking is close to free — add it |
| Real-time, high-QPS, tight latency SLA | Reranking cost is real; test aggressively, keep k small |
| Context window is generous and model handles long context well | Lower priority — the model can partially compensate by reading everything |
| Context window is tight, or you're paying per input token | Higher priority — every position wasted on a mediocre chunk is a lost opportunity for a good one |
A simple test before committing: hold out a sample of real queries with known-correct source chunks, measure precision@5 before and after reranking. If reranking moves ranks meaningfully — the right chunk lands in position 1-3 instead of 15-30 — the latency cost is almost always justified for anything beyond the tightest real-time constraints. If it barely moves the needle, your bottleneck is elsewhere, likely retrieval or chunking.
The Ordering Problem Doesn't End at Reranking
Even a well-reranked list has a second, subtler failure mode: research on long-context models (the "lost in the middle" findings) has repeatedly shown that models weight the start and end of a context window more heavily than the middle, regardless of how well things are ranked. Reranking gets the best candidates to the top of your list; how you then arrange them inside the prompt — front-loading the highest-scored chunks, deciding what to prune when the window is tight — is a second, separate decision the pipeline still has to make explicit. This is a large part of why teams that own a RAG system end up treating retrieval, reranking, and context assembly as four distinct product decisions rather than one monolithic "search" step.
Choosing a Reranker: Off-the-Shelf, Hosted, or Fine-Tuned
Most teams should start with an off-the-shelf, general-purpose cross-encoder before considering anything custom — the accuracy gap between a good general reranker and a domain fine-tuned one is usually smaller than the gap between having no reranker and having one at all. Cohere Rerank, Voyage AI's rerankers, and open cross-encoder checkpoints hosted through sentence-transformers are common starting points teams reach for.
- Hosted API rerankers (Cohere, Voyage, Jina) trade a per-call cost and network hop for zero infrastructure — reasonable for most teams below very high query volumes.
- Self-hosted open cross-encoders (e.g.
ms-marcofine-tunes,bge-rerankervariants) remove the network hop and per-call cost but require GPU capacity and batching discipline. - Domain fine-tuned rerankers are worth the investment only once you have enough labeled query-document relevance pairs from your own domain to meaningfully outperform a general model — usually a later-stage optimization, not a starting point.
Don't skip the baseline comparison: benchmark your actual retrieval + rerank pipeline against retrieval alone, on your own queries, before assuming any specific reranker is worth its cost. Generic benchmarks like BEIR are useful for picking a starting candidate, but they don't substitute for measuring against your own corpus and query distribution.
Where Prodinja Fits: Making Ordering a Choice, Not an Accident
Reranking only pays off if the ordering decision it produces actually reaches the model deliberately — many teams tune a reranker carefully, then let a hardcoded "top-5, in whatever order" assembly step quietly undo the work. Prodinja's Context Engineering tool is designed to make the ordering and pruning of retrieved context an explicit product choice rather than a silent default, so the reranking win you've earned upstream doesn't get lost in an unexamined assembly step downstream. It's a prototype experience aimed at making that layer visible and decidable, not a claim that it runs live inference against your production index today.
If reranking is one piece of a broader RAG architecture review, the complete guide to RAG and knowledge retrieval is a useful map of where this decision sits relative to chunking, embedding choice, and generation strategy.
Key Takeaways
- Reranking is a second stage, not a replacement — it re-scores the shortlist retrieval already found; it can't rescue documents retrieval missed entirely.
- Bi-encoders optimize for recall at scale; cross-encoders optimize for precision on a small candidate set — the two are complementary, not competing choices.
- The decision rule is diagnostic, not automatic: add a reranker when the right answer is in your retrieved set but not near the top, and your latency budget can absorb tens to hundreds of milliseconds.
- Narrow
kbefore reranking, not after — the candidate count you send to the cross-encoder is the main latency lever you control. - Measure
precision@5before and after on real queries before committing — don't add reranking on the assumption it will help. - Ordering doesn't stop at reranking — how chunks are arranged and pruned inside the final prompt is a separate decision that can undo a reranker's gains if left implicit.
- Start with an off-the-shelf cross-encoder before investing in domain fine-tuning; the gap from "none" to "general" is usually the bigger win.
Frequently Asked Questions
Does adding a reranker always improve RAG output quality?
Not always — reranking improves output quality only when the correct document is already somewhere in your retrieved candidate set but ranked low. If retrieval itself is missing the right chunks, reranking has nothing good to promote and won't fix a recall problem.
How much latency does a cross-encoder reranker typically add?
Reranking 50-100 candidates with a mid-sized cross-encoder typically adds tens to a few hundred milliseconds, depending on model size, hardware, and batching. The cost scales with the number of candidates reranked, which is why narrowing k before reranking matters more than the reranker choice itself.
What's the difference between a bi-encoder and a cross-encoder in RAG?
A bi-encoder encodes the query and document separately so document vectors can be precomputed and searched at scale; a cross-encoder encodes them jointly for a query-aware relevance score, at the cost of not being precomputable. Bi-encoders power fast first-stage retrieval; cross-encoders power precise second-stage reranking.
Can I skip reranking if I'm using a long-context model?
You can, but it's a tradeoff, not a free pass — long-context models still show measurable "lost in the middle" effects where content buried mid-context gets less effective attention than content near the start or end. A reranker that puts the best chunks first reduces how much you're relying on the model to compensate for poor ordering.
Is reranking worth it for a low-traffic internal tool versus a high-QPS production API?
For low-traffic, latency-tolerant use cases, reranking is close to a free precision win and worth adding by default. For high-QPS, tight-latency production APIs, test the actual precision gain against your specific latency SLA before committing, and keep the candidate count small to control cost.