Reciprocal Rank Fusion

Reciprocal rank fusion (RRF) combines several ranked lists of documents into one. In retrieval it is the usual way to merge, say, a BM25 ranking with a vector-search ranking, and it works from positions only: a list says that document A came first and document B third, and nothing about the scores behind those positions.

The formula

Cormack, Clarke, and Büttcher proposed RRF as an unsupervised fusion method. With ranks counted from 1, the fused score of document \(d\) over lists \(j=1,\dots,m\) is

\[ \mathrm{RRF}(d)=\sum_{j=1}^{m}\frac{1}{c+\mathrm{rank}_j(d)}, \]

where a document absent from list \(j\) contributes nothing for that list, and \(c\) is a constant that Cormack et al. set to 60 in their experiments. Documents are sorted by fused score. An optional weight \(w_j\) per list multiplies each term when one retriever should count for more.

Why rank-based fusion

BM25 scores are unbounded sums that depend on collection statistics and query length. Cosine similarities from an embedding model live in a narrow band such as 0.7 to 1.0. Adding the two raw numbers lets whichever scale is larger decide the result. The program below uses hypothetical scores for two lists and compares raw summation with RRF.

A=["d1","d2","d3","d4","d5"]   # lexical ranking
B=["d3","d6","d1","d7","d2"]   # dense ranking
def rrf(lists,c=60):
    sc={}
    for L in lists:
        for r,d in enumerate(L,1): sc[d]=sc.get(d,0)+1/(c+r)
    return sorted(sc.items(),key=lambda x:-x[1])
bm=dict(zip(A,[14.2,13.9,9.0,8.7,8.5])); de=dict(zip(B,[0.83,0.81,0.80,0.79,0.78]))
raw={d:bm.get(d,0)+de.get(d,0) for d in set(bm)|set(de)}
print("c = 60",[(d,round(s,4)) for d,s in rrf([A,B])][:3])
print("raw sum",sorted(raw.items(),key=lambda x:-x[1])[:4])

Output:

c = 60 [('d1', 0.0323), ('d3', 0.0323), ('d2', 0.0315)]
raw sum [('d1', 15.0), ('d2', 14.68), ('d3', 9.83), ('d4', 8.7)]

In the raw sum the BM25 numbers dominate: d3, ranked first by the dense retriever and third by BM25, falls to third place, and d4, which only the lexical list contains, is fourth. RRF sees d1 (ranks 1 and 3) and d3 (ranks 3 and 1) as symmetric and puts them in a tie at the top, and d2 (ranks 2 and 5) next. The fused list rewards documents that both retrievers place high, without needing the scores to be comparable.

The constant \(c\)

The constant sets how fast credit decays with rank. The weight of rank \(r\) relative to rank 1 is \((c+1)/(c+r)\). With \(c=1\), rank 10 earns 18% of what rank 1 earns, so the top positions dominate and a first-place finish in one list can outweigh moderate placements in several. With \(c=60\), rank 10 earns 87%, and ranks 1 to 20 are nearly equal, so the fused score mostly counts in how many lists a document appears. A large \(c\) therefore acts like agreement voting, and a small \(c\) keeps the top of each list decisive. In the same toy, \(c=1\) gives d1 and d3 0.75 each, d2 0.5, and d6, which only the dense list ranks second, 0.333. Elasticsearch exposes the constant under the name rank_constant with default 60.

Limits

• Ranks discard confidence. A list in which the top document is far ahead of the second looks the same as one where they are nearly tied.
• Equal weights give a weak retriever the same vote as a strong one. Weighted RRF or evaluation per query slice shows whether fusion helps for the queries you have.
• Only documents in the top \(n\) of a list score, so the depth \(n\) taken from each list affects the result, as does tie-breaking.
• Fusion does not remove duplicates by itself. Documents from different lists must share an identifier, or the same chunk is counted as two.
• It is a heuristic with good empirical behavior, not an estimate of relevance probability. A reranker that reads query and document together can reorder the fused candidates.

References

• Cormack, G. V., Clarke, C. L. A., Büttcher, S., Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR 2009 — http://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf