Mean Reciprocal Rank (MRR)

Mean reciprocal rank scores a ranked result list by one question: how far down the list is the first relevant item? It suits tasks where a single good hit is enough, such as finding the one passage that answers a factual question or the one correct file in a lookup.

Definition

For a query \(i\), let \(\mathrm{rank}_i\) be the position, counted from 1, of the first relevant result. The reciprocal rank is \(1/\mathrm{rank}_i\), or 0 if no relevant result appears (in the whole list, or in the top \(k\) for MRR@\(k\)). MRR is the average over the \(Q\) queries:

\[ \mathrm{MRR}=\frac{1}{Q}\sum_{i=1}^{Q}\frac{1}{\mathrm{rank}_i}. \]

The score is 1 when every query has a relevant result first, and it drops to 0.5 for a first hit at rank 2, 0.333 at rank 3, and so on.

A worked example

Four hypothetical queries return ranked lists in which a 1 marks a relevant result at that position.

runs={"q1":[0,0,1,0,0],"q2":[1,0,0,0,0],"q3":[0,0,0,0,0],"q4":[0,1,1,0,0]}
rr=lambda r:next((1/(i+1) for i,x in enumerate(r) if x),0)
for q,r in runs.items(): print(q,round(rr(r),3))
print("MRR",round(sum(rr(r) for r in runs.values())/len(runs),4))

Output:

q1 0.333
q2 1.0
q3 0
q4 0.5
MRR 0.4583

Query q1 has its first relevant result at rank 3, q2 at rank 1, q3 has none, and q4 has its first at rank 2 (the second relevant item at rank 3 is ignored). The mean is \((0.333+1+0+0.5)/4=0.4583\).

Properties worth knowing

Only the first relevant item counts. The program below compares system A, which places one relevant document first and then nothing relevant in positions 2 to 6, with system B, which places five relevant documents at positions 2 to 6.

A=[1,0,0,0,0,0]; B=[0,1,1,1,1,1]
print("MRR A",rr(A),"B",rr(B),"; relevant in top6 A",sum(A),"B",sum(B))

MRR A 1.0 B 0.5 ; relevant in top6 A 1 B 5

A has reciprocal rank 1.0 and B has 0.5, although A returns one relevant document in the top six and B returns five. If the application needs several relevant documents, for example to assemble evidence for an answer that draws on multiple passages, MRR hides the difference. Recall@\(k\) or nDCG measures it.

The weighting is steep at the top. Moving the first hit from rank 1 to rank 2 costs 0.5, while moving it from rank 9 to rank 10 costs 0.011. This matches users who read only the first result, and it understates the difference between a result at 10 and one at 50 when a reranker or a language model reads the entire top-\(k\).

It is an average of ratios, not of ranks. A query whose first hit is at rank 100 and one at rank 1000 both contribute almost nothing, so MRR is insensitive to how bad the bad cases are, and a mean of 0.5 can come from half the queries at rank 1 and half missing entirely, or from every query at rank 2. Look at the distribution of reciprocal ranks, or the fraction with a hit in the top \(k\), next to the mean.

It depends on the judgments. A relevant document that was never labeled counts as a non-hit, so an improved system can lose MRR. See Relevance Judgments and Pooling.

Report uncertainty. MRR is an average over queries, so with a few dozen queries it is noisy; compare two systems with a paired bootstrap over per-query reciprocal ranks.

Neighboring metrics

• Recall@\(k\): the fraction of relevant documents appearing in the top \(k\). It ignores order within \(k\).
• nDCG@\(k\): handles graded relevance and several relevant items, with a logarithmic position discount.
• Hit rate or success@\(k\): whether any relevant result appears in the top \(k\). MRR@\(k\) refines it by rewarding earlier hits.