Reranking
In one sentence
Reranking is a second pass that reorders retrieved results by relevance using a slower and more accurate model than the initial search, and it is what fixes the near-misses fast retrieval gets wrong.
Not to be confused with Chunking.
Definition
Reranking is a second look at what a search returned. A slower, more accurate model reads the question and each candidate passage together, then puts the best one first.
The first search is built for speed. Reranking is what catches the passage that was close but not right.
The two stages run on fundamentally different model architectures, and that difference is the entire reason the two-stage approach works.
Bi-encoder retrieval, the first stage
- The query and every document are embedded independently, then compared by how close the vectors sit.
- Document embeddings are computed in advance, so search is extremely fast and scales to a large corpus.
- The cost is that the query and the document never interact during encoding. The model is never given the chance to consider them together.
Cross-encoder reranking, the second stage
- The query and one candidate passage are fed through the model together, so attention operates across both, and the model scores relevance from that interaction directly.
- Considerably more accurate. Also far more expensive, because it needs one model pass per candidate and nothing can be precomputed.
- That expense is why it runs over a shortlist of perhaps twenty to fifty candidates rather than over the whole corpus.
The standard pipeline shape
- Retrieve broadly, perhaps the top fifty by vector similarity, favoring recall over precision.
- Rerank that shortlist with a cross-encoder, favoring precision.
- Pass the top three to five reranked results to the language model.
- The logic is that the first stage must not miss the right answer, and the second stage must not let a wrong one reach the model.
What reranking cannot fix
- If the right chunk was never retrieved in the first stage, reranking cannot surface it. It improves ordering within the shortlist, never the contents of the shortlist.
- If the right answer does not exist in the corpus at all, reranking will confidently order a set of wrong answers.
Common misconception
That reranking is an optional refinement for teams chasing marginal gains. In a corpus full of passages that look alike, which describes most business documentation, it is often the difference between the right answer and a plausible neighbor. Tiered pricing is the canonical case: five passages a bi-encoder finds nearly identical and a cross-encoder separates without difficulty.
Why it matters commercially
Reranking is the cheapest quality improvement available to a RAG system that keeps returning nearly-right answers. If the symptom is that it found something related but not quite right, reranking is the correct intervention.
In voice specifically
Reranking spends time inside the time-to-first-audio budget, and a cross-encoder pass over fifty candidates is not free. The practical mitigations are to rerank fewer candidates, use a smaller reranker, or run reranking only when the initial retrieval scores came back ambiguous. That last one is the elegant option: spend the latency only when the retrieval was uncertain.
Where AsqVox fits
Business documents with tiered plans, regional variations and policy sections that read alike are exactly the corpus type where reranking earns back the latency it costs. That is the corpus shape the Orb's true RAG over uploaded documents has to work against.
Visual
Fast search finds five candidates. One of them is right.
Recall first, precision second. Both, or neither works.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
No reliable cross-industry figure exists for the retrieval accuracy improvement attributable to reranking. Published gains come from specific models on specific datasets and do not transfer directly.
-no reliable figureAny percentage uplift quoted at you should be treated as dataset-specific. The honest version of the claim is architectural rather than numerical, and it is the row below.
Cross-encoder architectures rank relevance more accurately than bi-encoder architectures, and cost materially more per comparison. That difference is why the two-stage pipeline exists at all.
Architectural factindustry rangeEstablished retrieval architecture, 2026 - This can be stated with confidence because it follows from how the two architectures work, not from a measurement. It is the claim to make when a vendor wants a percentage instead.
Reranking is applied to shortlists rather than to whole corpora, because the per-candidate cost rules out exhaustive application. Twenty to fifty candidates is the usual shortlist.
20 to 50 candidatesindustry rangeStandard pipeline convention, 2026 - An engineering convention rather than a published finding. The shortlist size is the main dial available when reranking latency needs to come down.
Sub-second time to first audio is the design target, and reranking sits inside that budget.
Under 1,000 msindustry rangeIndustry working threshold, 2026 - A working convention, not a published standard. It is what makes conditional reranking, triggered only when retrieval scores are ambiguous, an architecturally sensible pattern in voice specifically.
Past roughly 1,500 milliseconds, a conversation is widely reported to feel broken. Reranking latency counts against that total.
1,500 msindustry rangeIndustry working threshold, 2026 - A shared working line rather than a measured cliff. A reranker that improves accuracy and pushes the total past it has not improved the product.
Examples
In practice
A voice agent serving a company with five pricing tiers across three regions retrieves a plausible but wrong tier roughly a fifth of the time. All fifteen combinations produce near-identical embeddings. Adding a cross-encoder reranker over the top twenty candidates largely resolves the confusion at a cost of roughly 120 milliseconds. The team then makes reranking conditional on the top two retrieval scores sitting within a narrow margin, which recovers most of the latency and keeps the accuracy gain.
The everyday version
Reranking is a second look. The quick search hands back five passages that all talk about pricing. A slower second check then reads each one against the actual question and works out which passage is genuinely the answer. Skip that check and the system will sometimes settle on the one about the wrong plan.
Usage
Who says it
- Search and AI engineers, precisely.
- It appears in platform documentation and, increasingly, in vendor technical differentiation.
- Almost never in a buyer conversation, though "it checks its answer before it speaks" is a fair buyer-facing translation.
Where it turns up
- Next to retrieval architecture, hybrid search, the reranking model, and retrieval latency.
- In a technical evaluation, whether reranking is available and configurable is a reasonable proxy for how seriously a platform treats retrieval quality.
Common misuse
- Expecting reranking to fix a recall problem. It reorders the shortlist and cannot add to it.
- Applying reranking unconditionally in a latency-sensitive voice path without measuring what it costs.
- Treating it as optional in a corpus full of similar passages, which is exactly where it matters most.
Questions people ask
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder embeds the query and each document separately and compares the vectors, which is fast because the document side is computed in advance. A cross-encoder puts the query and one candidate passage through the model together, so attention runs across both and the score reflects how they interact. The cross-encoder is considerably more accurate and far more expensive, which is why it runs over a shortlist of twenty to fifty candidates rather than the whole corpus.
Does reranking fix bad retrieval?
Only the ordering part of it. Reranking reorders the shortlist the first stage produced. It cannot add a passage that never made the shortlist, and it cannot invent an answer the corpus does not contain. If the right chunk is never retrieved, the fix is chunking or recall, not reranking.
How much latency does reranking add to a voice agent?
Enough to matter, because it sits inside the time to first audio budget, where sub-second is the working target and past roughly 1,500 milliseconds a conversation is widely reported to feel broken. In one worked case a cross-encoder over twenty candidates cost about 120 milliseconds. The usual mitigations are fewer candidates, a smaller reranker, or reranking only when the first-stage scores were ambiguous.
How much does reranking improve accuracy?
There is no reliable cross-industry figure. Published gains come from specific models on specific datasets and do not transfer to your corpus, so any percentage uplift quoted at you should be treated as dataset-specific. What can be stated is architectural: cross-encoders rank relevance more accurately than bi-encoders, at a materially higher cost per comparison, and that is why the two-stage pipeline exists.
Last reviewed 30 July 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.