Mean Average Precision (MAP)

Mean average precision evaluates ranked results, one query at a time. For a single query, average precision records the precision at each rank where a relevant item appears and averages those values; MAP is the mean of those averages across queries.

\(\text{MAP} = \dfrac{1}{Q}\sum_{q=1}^{Q}\text{AP}(q), \qquad \text{AP} = \dfrac{1}{R}\sum_{k=1}^{N} P(k)\cdot \text{rel}(k)\)

with \(N\) retrieved items, \(P(k)\) the precision over the top \(k\), and \(\text{rel}(k)\) equal to 1 when the item at rank \(k\) is relevant. The denominator \(R\) is the one to read carefully.

R counts every relevant item, not every retrieved one

\(R\) is the number of relevant items that exist for the query, including any the system never returned. Those contribute nothing to the numerator while still enlarging the denominator, so a missed document costs score. That is how AP charges for recall without a separate recall term.

import numpy as np

def average_precision(relevant, n_relevant_total):
    """IR average precision: mean of precision@k at each rank holding a relevant item,
       divided by every relevant item there is — including any never retrieved."""
    hits = 0
    total = 0.0
    for k, is_relevant in enumerate(relevant, start=1):
        if is_relevant:
            hits += 1
            total += hits / k
    return total / n_relevant_total

queries = {
    "q1: relevant at 2 and 4": ([0, 1, 0, 1, 0], 2),
    "q2: both at the top":     ([1, 1, 0, 0, 0], 2),
    "q3: one of three missed": ([1, 0, 1, 0, 0], 3),
}
scores = []
for label, (relevant, total) in queries.items():
    ap = average_precision(relevant, total)
    scores.append(ap)
    print(f"{label:26} AP {ap:.4f}")
print(f"MAP over {len(scores)} queries          {np.mean(scores):.4f}")
truncated = average_precision([0, 1, 0, 1, 0][:3], 2)
print(f"q1 truncated to the top 3 (AP@3)  {truncated:.4f}")
# q1: relevant at 2 and 4    AP 0.5000
# q2: both at the top        AP 1.0000
# q3: one of three missed    AP 0.5556
# MAP over 3 queries          0.6852
# q1 truncated to the top 3 (AP@3)  0.2500

The first query returns relevant items at ranks 2 and 4, giving precisions of 1/2 and 2/4 and an AP of 0.5. The second returns both at the top — precisions of 1/1 and 2/2 — and scores 1.0. The third finds two of three relevant items, at ranks 1 and 3, so the numerator is \(1 + 2/3\) but the denominator is 3, and the AP is 0.5556 rather than the 0.8333 that dividing by the two found items would give.

MAP over the three is 0.6852. Note what that averaging does: each query contributes equally regardless of how many relevant documents it has, so a query with one relevant item counts as much as one with fifty. Where queries differ wildly in difficulty, the per-query distribution is worth looking at before the mean.

In practice it is truncated

Real systems return ten results, not all of them, and the metric is computed over that cut: AP@k, and MAP@k across queries. The last line shows why the cut has to be stated — the first query’s AP is 0.5 over five results and 0.25 over three, because the relevant item at rank 4 has disappeared while \(R\) stays at 2.

A MAP quoted without its \(k\) is not comparable with another. MAP@10 and MAP@100 on the same system are different numbers, and the second is always at least as large.

Three metrics share this name

The three are close in spirit and different in arithmetic, so numbers do not carry across. A detection paper’s mAP and a search team’s MAP are not on the same scale.

Where it is used

MAP runs from 0 to 1, with 1.0 meaning every relevant item was returned above every irrelevant one for every query. It suits search and recommendation, where what matters is finding relevant items early and the user sees a ranked list rather than a decision. TREC and Kaggle leaderboards report it as standard.

What it does not measure is how many relevant items exist or how graded their relevance is — every relevant item counts the same. Where relevance comes in degrees, a gain-based measure such as NDCG is the better fit.

References

• scikit-learn API reference, average_precision_score
• Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn and Andrew Zisserman, “The Pascal Visual Object Classes (VOC) Challenge”, International Journal of Computer Vision, 88, 2010, 303–338
• COCO, Detection Evaluation