Cross-Encoder

A cross-encoder is a model that reads a query and a candidate document together, as one input, and outputs a single relevance score. It is usually a transformer fed the concatenation “query [separator] document”, with a small head that turns the final representation into a number. It is contrasted with the bi-encoder (two-tower) design used for embedding retrieval, where query and document are encoded separately and compared afterward.

Joint scoring versus separate embeddings

A bi-encoder maps each text to a vector independently, and relevance is a similarity such as the dot product of the two vectors: \( \text{score}(q,d)=f(q)\cdot g(d) \). Because \(g(d)\) does not depend on the query, every document vector can be computed once, stored in an index, and searched with approximate nearest neighbor methods. The price is that the whole content of a document must be squeezed into a fixed-size vector before anyone knows what will be asked.

A cross-encoder computes \(\text{score}(q,d)=h(q\,\Vert\,d)\). Self-attention runs across the query tokens and the document tokens, so the model can see that the query asks for the limit for contractors and that the passage’s qualifier says “employees only”. Interactions of that kind (negation, qualifiers, which entity a number belongs to, matching a specific identifier) are where cross-encoders gain accuracy over embeddings.

Why it is used as a reranker and not as a retriever

The accuracy comes with a cost structure: no part of the work can be done ahead of time, because the input contains the query. Scoring one query against a corpus of \(N\) documents takes \(N\) forward passes. A hypothetical comparison with one million documents of 200 tokens and an 8-token query:

• Scoring the whole corpus with a cross-encoder needs 1,000,000 forward passes of 208 tokens each, about \(2.08\times10^{8}\) tokens through the model for every query.
• Reranking the top 20 candidates from a first-stage retriever needs 20 passes, 4,160 tokens. The top 100 needs 100 passes, 20,800 tokens.

So the standard design is a funnel: a cheap retriever (lexical, dense or both, see Hybrid Search) produces tens to hundreds of candidates, and the cross-encoder reorders only those. Nogueira and Cho showed an early version of this on MS MARCO with a BERT model that scores query-passage pairs and reorders a first-stage list, and the BEIR benchmark by Thakur et al. reports that reranking models reach strong zero-shot results at high computational cost.

Training and output

A cross-encoder is trained on labeled pairs: queries with relevant and non-relevant documents, using a pointwise loss (binary relevance), a pairwise loss (relevant document scores above a non-relevant one), or a listwise loss. Its output is a score that is meaningful for ordering the candidates of one query. It is not automatically a probability, and scores for different queries are not comparable unless calibrated against labeled examples. A threshold such as “abstain if the best score is below 0.3” needs that calibration.

Conditions and failure modes

• A ceiling from the first stage. A reranker can only reorder what it receives. If the relevant chunk is not among the candidates, no reranker finds it, so candidate-pool recall should be reported separately from the final ordering.
• Depth is not free. More candidates cost more compute and give a noisy reranker more chances to promote a wrong document. When score errors are comparable to the gaps, expanding the pool can promote a wrong document above a correct one. Depth can therefore have diminishing or negative returns even as candidate recall improves. Choose the depth by sweeping it on a benchmark.
• Input length. Query plus document must fit the model’s context, so long passages are truncated, and a decisive sentence beyond the cut cannot influence the score.
• Domain shift. A cross-encoder trained on web search may mis-rank code, identifiers or specialized text, which is measured per query type.
• Multi-part queries. A single joint score for a query with two intents can favor a passage that matches one of them well. Splitting the query beforehand can help.

Related terms

A late-interaction model such as ColBERT keeps per-token document vectors so part of the computation is precomputed, which places it between bi-encoder and cross-encoder in cost. An LLM-based listwise reranker takes the whole candidate list in one prompt and produces an order, trading per-candidate passes for one long prompt.

References

• Nogueira and Cho, Passage Re-ranking with BERT — https://arxiv.org/abs/1901.04085
• Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — https://arxiv.org/abs/2104.08663