AUC (Area Under the Curve)

AUC means “area under the curve” and does not say which curve. Two are in common use and they are not interchangeable:

The baselines are why the ambiguity costs something. An AUC of 0.6 is weak if it means AUROC and outstanding if it means PR-AUC on a dataset with 1 percent positives. Unqualified, “AUC” conventionally means AUROC, and the convention is worth stating rather than assuming.

Two computations, one number

AUROC can be computed two ways that look unrelated. Geometrically: sweep the threshold, collect \((\text{FPR}, \text{TPR})\) points, and take the trapezoidal area. By ranking: for every positive–negative pair, score 1 if the positive is higher, 0.5 if tied, 0 otherwise, and divide by the number of pairs.

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve

def pairwise_auc(y, s):
    """Count every positive-negative pair; a tie is half a win."""
    pos, neg = s[y == 1], s[y == 0]
    wins = np.sum(pos[:, None] > neg[None, :]) + 0.5 * np.sum(pos[:, None] == neg[None, :])
    return wins / (pos.size * neg.size)

def geometric_auc(y, s):
    """Trapezoidal area under the ROC curve."""
    fpr, tpr, _ = roc_curve(y, s)
    return float(np.sum(np.diff(fpr) * (tpr[1:] + tpr[:-1]) / 2))

small_y = np.array([1, 0, 1, 0, 1, 0])
small_s = np.array([0.92, 0.85, 0.70, 0.60, 0.30, 0.20])
print(f"six cases   pairwise {pairwise_auc(small_y, small_s):.4f}  "
      f"geometric {geometric_auc(small_y, small_s):.4f}  sklearn {roc_auc_score(small_y, small_s):.4f}")

rng = np.random.default_rng(31)
y = np.concatenate([np.ones(300), np.zeros(700)])
s = np.concatenate([rng.normal(1, 1, 300), rng.normal(0, 1, 700)])
print(f"continuous  pairwise {pairwise_auc(y, s):.4f}  "
      f"geometric {geometric_auc(y, s):.4f}  sklearn {roc_auc_score(y, s):.4f}")

coarse = np.round(s * 2) / 2
print(f"rounded     pairwise {pairwise_auc(y, coarse):.4f}  "
      f"geometric {geometric_auc(y, coarse):.4f}  ties {int(np.sum(coarse[y==1][:,None] == coarse[y==0][None,:]))}")
print(f"Gini for the continuous scores: {2 * roc_auc_score(y, s) - 1:.4f}")
# six cases   pairwise 0.6667  geometric 0.6667  sklearn 0.6667
# continuous  pairwise 0.7694  geometric 0.7694  sklearn 0.7694
# rounded     pairwise 0.7640  geometric 0.7640  ties 22880
# Gini for the continuous scores: 0.5388

They agree to four decimal places, which is not a coincidence — it is the identity Fawcett describes, the one that also makes AUROC equivalent to the Mann–Whitney U statistic and the Wilcoxon rank-sum test. The first line is the small worked case: three positives at 0.92, 0.70 and 0.30 against three negatives at 0.85, 0.60 and 0.20 win 6 of the 9 pairs, so the area is 6/9.

The third line shows what ties do. Rounding the same scores to the nearest 0.5 creates 22,880 tied pairs, each counted as half a win, and the value moves from 0.7694 to 0.7640. Here the ties were mostly between correctly ordered pairs, so the value falls. The direction is not fixed: collapsing a wrongly ordered pair credits it half a win and raises the area. What is reliable is that coarse scores discard information the metric would otherwise use. This is worth knowing for vote-share outputs and rating scales, where ties are the norm rather than an edge case.

The last line is the Gini coefficient, \(2 \times \text{AUROC} - 1\), which credit scoring reports in place of the area. It carries exactly the same information on a scale where no skill is 0.

AP is not an interpolation

On the precision–recall side there are again two computations, and here they genuinely disagree. Average precision is a weighted sum of the precision at each threshold:

\(\text{AP} = \sum_n (R_n - R_{n-1})\,P_n\)

AP is frequently described as a particular interpolation of the PR curve. It is the opposite: scikit-learn documents AP as “not interpolated and … different from computing the area under the precision-recall curve with the trapezoidal rule, which uses linear interpolation and can be too optimistic”. The interpolation belongs to the trapezoid, not to AP, and Davis and Goadrich give the reason it is a problem — “in PR space it is incorrect to linearly interpolate between points”, because the straight line between two PR points is not achievable.

So average_precision_score and auc(recall, precision) return different numbers on the same predictions, and a reported “PR-AUC” is ambiguous until you say which.

Where an area hides things

• It averages over thresholds you will not use. Two models with equal area can differ sharply where you operate — see ROC-AUC, or restrict the range with partial AUC.
• It says nothing about calibration. Any strictly increasing rescaling leaves it unchanged; a non-strict one such as isotonic regression creates ties, which can move it slightly, so a badly miscalibrated model can have a perfect area.
• A threshold does not carry over between score scales. Fawcett notes that “classifier scores should not be compared across model classes” — the areas of two models can be compared, but a single shared threshold cannot be applied to both.
• It has sampling error. Whether a difference of 0.01 is real depends on the sample size and on how correlated the two estimates are, since they are usually computed on the same cases. Use DeLong’s test or a paired bootstrap on the difference rather than comparing two separate intervals.

A complete report says which curve, which computation for PR, which averaging for multiclass — one-versus-rest or one-versus-one, macro or micro — the prevalence if any precision-side number appears, and an interval. That is more words than “AUC = 0.87”, and it is the difference between a number someone can act on and one they have to ask about.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Jesse Davis and Mark Goadrich, “The Relationship Between Precision-Recall and ROC Curves”, ICML 2006
• scikit-learn API reference, average_precision_score