ROC-AUC

ROC-AUC is the area under the ROC curve, a single number summarizing a classifier’s ranking across every threshold at once. It is often written AUROC to distinguish it from the area under a precision–recall curve.

Its value comes from an interpretation that has nothing obvious to do with area. Fawcett states it directly: “the AUC of a classifier is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance. This is equivalent to the Wilcoxon test of ranks.” An AUROC of 0.85 means that if you draw one positive and one negative at random, the positive gets the higher score 85 percent of the time.

The full statement needs a tie term:

\(\text{AUROC} = P(s^+ > s^-) + \tfrac{1}{2}P(s^+ = s^-)\)

which matters whenever scores are coarse — a vote share out of 100 trees, a rounded probability, a rating scale. Ties are scored as half a win. That moves the value toward 0.5 when the tied pairs were correctly ordered, and away from it when they were not — merging a badly ordered pair can raise the area, for instance from 0.75 to 0.875. The direction depends on which pairs were collapsed.

Reading the value

Bands like 0.7–0.8 “acceptable” and 0.8–0.9 “good” circulate widely. They are convention rather than fact, and the useful comparison is against what is achievable in your domain: 0.65 is strong for some problems and unusable for others.

What the area averages over

ROC-AUC integrates across the whole false positive range, which means it gives equal weight to an FPR of 0.9 — a threshold nobody would deploy — and to an FPR of 0.01. Two models can reach the same area by being good in different places.

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve

def tpr_at_fpr(y, s, limit):
    """Recall reachable while holding the false positive rate at or below limit."""
    fpr, tpr, _ = roc_curve(y, s)
    return float(tpr[fpr <= limit].max())

rng = np.random.default_rng(21)
n_pos, n_neg = 5000, 50000
y = np.concatenate([np.ones(n_pos), np.zeros(n_neg)])
negatives = rng.normal(0.0, 1.0, n_neg)
u, a, b = rng.uniform(size=n_pos), rng.normal(0, 1, n_pos), rng.normal(0, 1, n_pos)

models = {
    "steady": np.concatenate([1.0 + rng.normal(0, 1, n_pos), negatives]),
    "sure about a fifth": np.concatenate([np.where(u < 0.20, 4.2 + 0.6 * a, 0.75 + b), negatives]),
}
for name, s in models.items():
    print(f"{name:19} AUC {roc_auc_score(y, s):.4f}  "
          f"recall at FPR<=0.01 {tpr_at_fpr(y, s, 0.01):.4f}  "
          f"at FPR<=0.10 {tpr_at_fpr(y, s, 0.10):.4f}")
# steady              AUC 0.7609  recall at FPR<=0.01 0.1050  at FPR<=0.10 0.3936
# sure about a fifth  AUC 0.7608  recall at FPR<=0.01 0.2644  at FPR<=0.10 0.4524

The areas are 0.7609 and 0.7608, the same to three decimal places. At a false positive rate of 1 percent — a realistic budget for anything that generates work — the first model catches 10.5 percent of positives and the second catches 26.4 percent. Choosing by AUROC here is choosing at random between a model and one that finds two and a half times as much.

The same issue appears whenever two ROC curves cross: the areas can be equal while each model is the better choice in a different region. So an AUROC comparison answers “which ranks better on average”, and the question usually being asked is “which is better where we operate”.

Under heavy imbalance

Invariance to class balance is genuinely useful and is also where AUROC flatters a model. The false positive rate divides by \(FP + TN\). With 100,000 negatives and 100 positives, an FPR of 0.01 is 1,000 false alarms against at most 100 real ones — a curve that looks strong and an alert queue that is 90 percent noise.

Saito and Rehmsmeier trace the problem to reading rather than to the metric, attributing it to “an intuitive but wrong interpretation of specificity”. Davis and Goadrich add a result that matters when AUROC is used as a training objective: “algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve”. Optimizing for the summary you are not going to report is a real risk.

What to report with it

• Recall at a workable false positive rate. One number, directly comparable, and immune to the averaging problem above.
• The area over the region you use, rather than all of it — see partial AUC.
• A precision-side summary when positives are rare — PR-AUC or average precision, reported next to the prevalence that is their baseline.
• An interval on the difference. Whether a gap of 0.01 is real depends on the sample size and on the covariance between the two estimates, which share the evaluation set. DeLong’s test or a paired bootstrap answers it; two separately reported intervals do not.
• Calibration, separately. AUROC is unchanged by any strictly increasing rescaling, so it cannot see whether 0.9 means 0.9 — that is calibration.

For multiclass problems, state the scheme: one-versus-rest averaged unweighted or by support, or one-versus-one pairwise. They are different numbers. The naming itself is ambiguous enough to be worth pinning down — see AUC.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Jesse Davis and Mark Goadrich, “The Relationship Between Precision-Recall and ROC Curves”, ICML 2006
• Takaya Saito and Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 10(3), 2015