Reranking

Reranking is a second stage of retrieval that takes the candidates found by a fast first search and rescores them against the query using another scoring method, often a costlier model, to choose a final subset. Whether this improves relevance must be measured. It is common in search engines and in retrieval-augmented generation, where only a handful of passages fit in the model’s context.

Why two stages

A first-stage search, by keywords or embeddings, searches a search index efficiently, often using postings or approximate vector search to avoid comparing every record: an embedding search compares vectors computed separately for each. A cross-encoder reranker reads a query and candidate together to estimate relevance. Other reranking methods use different signals; no method guarantees that the selected passage answers the question. Google’s documentation for its ranking service draws this contrast: compared with embeddings, which look only at semantic similarity, a ranking model scores “how well a document answers a given query.” Reading pairs together is too expensive to do for every passage in a corpus, so it is applied only to a shortlist, typically a few dozen to a hundred candidates, from which the top few are kept.

Rerankers vary in cost. Google RAG Engine’s comparison, as of September 2026, lists a dedicated ranking model at under 100 milliseconds per request and using a general language model as the reranker at one to two seconds. These are product comparison figures, not a p95 or service guarantee for a particular workload. Measure candidate count, passage length, concurrency, and end-to-end latency against the available budget. Also check per-record input limits: Google Ranking API models through version 003 allow 512 tokens, and version 004 and later allow 1,024, counting title and content together. Excess text is truncated. Preserve relevant conditions in the actual ranker input, using its tokenizer rather than assuming the generator’s count applies.

What it can and cannot fix

Reranking reevaluates ordering and can make it better or worse. If an answering passage moves from seventh to first, it becomes available in a final top-three context; whether the answer is then correct still requires generation evaluation. Increasing the cutoff from three to five alone does not include an unchanged seventh-place passage. Rescoring cannot recover a passage absent from the candidate pool. Compare candidate coverage with coverage at a fixed final cutoff, then evaluate the answer. Reordering a full pool leaves its membership unchanged, so membership before and after that permutation cannot measure ranking quality.

A reranker’s score measures relevance to the query, not truth, currency, or permission. A superseded policy is highly relevant to a question about that policy. Filters on effective date, audience, and access rights belong before ranking, not after it. The scoring model behind cross-encoder rerankers, and how multiple candidate lists are fused, are explained in Retrieval-Augmented Generation and Tool-Using Agents; where reranking sits among the causes of a wrong answer is covered in Last Year’s Policy, Cited Correctly.

References:

• Google Cloud, Improve search and RAG quality with ranking API (Creative Commons Attribution 4.0).
• Google Cloud, Reranking for RAG Engine (Creative Commons Attribution 4.0) — reranker latency comparison.