ROC Curve (Receiver Operating Characteristic)

The ROC curve shows what a binary classifier can do at every possible threshold, rather than at the one somebody happened to pick. Each threshold \(\tau\) — predict positive when the score is at least \(\tau\) — produces one pair of rates:

• True positive rate, \(TP/(TP+FN)\). This is recall, also called sensitivity.
• False positive rate, \(FP/(FP+TN)\), sometimes called fall-out. It is one minus specificity.

Plot \((\text{FPR}(\tau), \text{TPR}(\tau))\) and sweep \(\tau\) from above every score down to below every score, and the point traces a path from \((0,0)\) to \((1,1)\). In practice you sort the cases by score and walk down the list, updating the counts as each case is admitted as a positive, which makes the curve a staircase rather than a smooth arc.

Two things that do not change it

Only the ordering of the scores enters that construction, and each axis is computed inside a single column of the confusion matrix. Both facts have consequences worth demonstrating.

import numpy as np
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(12)
pos_scores = rng.beta(5, 2, 2000)
neg_scores = rng.beta(2, 5, 40000)
y = np.concatenate([np.ones(pos_scores.size), np.zeros(neg_scores.size)])
s = np.concatenate([pos_scores, neg_scores])

draws = 2_000_000
wins = pos_scores[rng.integers(pos_scores.size, size=draws)] > neg_scores[rng.integers(neg_scores.size, size=draws)]
print(f"roc_auc_score          {roc_auc_score(y, s):.4f}")
print(f"P(random + beats -)    {wins.mean():.4f}")

squashed = 1 / (1 + np.exp(-10 * (s - 0.5)))
print(f"after monotone squash  {roc_auc_score(y, squashed):.4f}")

for n_neg in (2000, 40000):
    keep = np.concatenate([np.arange(pos_scores.size), pos_scores.size + np.arange(n_neg)])
    print(f"negatives {n_neg:5d}  prevalence {pos_scores.size/(pos_scores.size+n_neg):.4f}  "
          f"AUC {roc_auc_score(y[keep], s[keep]):.4f}")
# roc_auc_score          0.9598
# P(random + beats -)    0.9599
# after monotone squash  0.9598
# negatives  2000  prevalence 0.5000  AUC 0.9588
# negatives 40000  prevalence 0.0476  AUC 0.9598

The first two lines confirm the interpretation that makes AUC worth reporting. Fawcett states it as a property: “the AUC of a classifier is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance. This is equivalent to the Wilcoxon test of ranks.” Drawing two million random pairs gives 0.9599 against the integrated 0.9598.

The third line squashes every score through a steep sigmoid — a drastic change to the values, no change at all to their order — and AUC is identical to four decimal places. Any strictly increasing rescaling, including converting logits to probabilities or applying a calibrator, leaves the ROC curve exactly where it was.

The last two lines change the prevalence by a factor of ten and the AUC does not follow, which is the property Fawcett highlights: “ROC curves have an attractive property: they are insensitive to changes in class distribution.” This holds for a population change in the mix alone with unchanged class-conditional score distributions. Resampling negatives can change a finite-sample curve through sampling variation, as the last two estimates illustrate; the population curve remains the same. It does not hold when the classes themselves change — a new fraud pattern, a different patient population — and it says nothing about carrying a training-set curve into production.

Reading the plot

What you see	What it means
The diagonal \(\text{TPR} = \text{FPR}\)	No skill. A coin flip weighted to fire this often would do as well
Bulging toward the top left	High recall reached at low cost in false alarms. More bulge is better
A point below the diagonal	Worse than chance at that threshold, which usually means an inverted label or sign
AUC near 0.5	The scores rank no better than chance on average over all pairs. A curve that crosses the diagonal can average to 0.5 and still have a genuinely useful operating point, so read the curve before discarding the model
AUC below 0.5	Almost always a bug rather than a finding — flipping the scores would beat it

Keep the curve and its summary distinct: ROC is the curve, ROC-AUC is the single number under it. Two curves that cross can have the same area while being the better choice in different regions, so the area alone can hide the part of the range you actually operate in.

Choosing a point on it

The curve says what is achievable; it does not say which point to take. That needs costs and prevalence.

Youden’s \(J = \text{TPR} - \text{FPR}\) picks the point furthest above the diagonal. It weights sensitivity and specificity equally, which is not the same as equal per-case costs: it is the iso-cost tangent of slope 1, whereas equal costs give a slope of \((1-\pi)/\pi\) — at a prevalence of 0.1 that is 9, not 1. Youden is the right target when the two rates matter equally, which is a class-balanced objective rather than a cost-neutral one. When they are not, the expected cost at a point is proportional to \((1-\pi)\,c_{FP}\cdot\text{FPR} + \pi\,c_{FN}\cdot(1-\text{TPR})\), and lines of equal cost in ROC space have slope

\(m = \dfrac{(1-\pi)\,c_{FP}}{\pi\,c_{FN}}\)

The best reachable operating point is where the lowest such line touches the curve. Fawcett draws the general conclusion from this: “a classifier is potentially optimal if and only if it lies on the convex hull” of the points in ROC space — the ROC convex hull, which also covers mixtures of two classifiers.

Because the curve is invariant to prevalence but the slope above is not, a shift in the positive rate after deployment leaves the curve intact and moves the right point along it. The curve does not need recomputing; the threshold does — see recalibrate thresholds.

Where the invariance becomes a blind spot

Not noticing class balance is useful when comparing scorers and misleading when reporting what a deployed system will do. The false positive rate divides by \(FP + TN\), and when negatives outnumber positives a thousand to one, an FPR of 0.01 is a tiny-looking number that means ten false alarms for every real positive. The curve looks excellent; the alert queue is unusable.

Saito and Rehmsmeier trace this to how the plot is read rather than to the plot itself: on imbalanced data, ROC’s visual interpretability “can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity”. Their recommendation is to add a precision–recall view, where the baseline is the prevalence and the cost of false alarms is visible — see precision.

Two further limits. The ROC curve says nothing about whether the scores are probabilities: a model whose reliability curve is badly wrong can have a perfect ROC, since only the ordering matters — that is what calibration measures. And Fawcett warns that “classifier scores should not be compared across model classes”, since one model may produce scores in [0, 1] and another in [-1, +1]; comparing them at a shared threshold is meaningless even though comparing their curves is not.

A single point, worked

Suppose a test set contains 100 positives and 900 negatives — a prevalence of 10 percent — and at some threshold you measure \(TP = 45\), \(FN = 55\), \(FP = 90\), \(TN = 810\). Then \(\text{TPR} = 45/100 = 0.45\) and \(\text{FPR} = 90/900 = 0.10\), so \((0.10, 0.45)\) is one point on the curve. Moving the threshold produces the rest of it.

For multiclass problems, compute one curve per class against all others and average the areas — unweighted for macro, by support for weighted — or pool the counts first for a micro average. Fawcett notes that the one-versus-rest formulation “compromises one of the attractions of ROC graphs, namely that they are insensitive to class skew”, because each negative set is a union of the other classes and their relative prevalences can shift the curve.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Takaya Saito and Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 10(3), 2015
• scikit-learn API reference, roc_auc_score