One-vs-Rest (OvR) AUROC

AUROC is defined for two classes. One-versus-rest turns a \(K\)-class problem into \(K\) binary ones: for each class \(i\), label it positive, label every other class negative, and score the model’s \(i\)-th output column as an ordinary binary problem.

\(\text{AUROC}_{\text{OvR}}(i) = \text{AUROC}(\text{class}_i \text{ vs rest})\)

That gives \(K\) numbers. They can be reported as they are — which is usually the more useful thing — or averaged into macro AUROC. With per-class values of 0.83, 0.76 and 0.70, the macro figure is 0.763.

What is in the “rest”

Each of those \(K\) numbers answers a different question, because each has a different negative set. That is easy to lose sight of when they are printed in a column.

import numpy as np
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(11)
classes, n = 4, 4000
y = rng.choice(classes, size=n, p=[0.6, 0.25, 0.1, 0.05])
scores = rng.normal(size=(n, classes))
for k in range(classes):
    scores[y == k, k] += 1.2
probs = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)

for k in range(classes):
    positives = (y == k).astype(int)
    auc = roc_auc_score(positives, probs[:, k])
    rest = np.bincount(y[y != k], minlength=classes)
    print(f"class {k}  support {positives.sum():4d}  AUROC {auc:.4f}  "
          f"rest is {' + '.join(f'{c}x class {j}' for j, c in enumerate(rest) if c)}")
# class 0  support 2436  AUROC 0.8517  rest is 1010x class 1 + 376x class 2 + 178x class 3
# class 1  support 1010  AUROC 0.8494  rest is 2436x class 0 + 376x class 2 + 178x class 3
# class 2  support  376  AUROC 0.8602  rest is 2436x class 0 + 1010x class 1 + 178x class 3
# class 3  support  178  AUROC 0.8701  rest is 2436x class 0 + 1010x class 1 + 376x class 2

Four classes, all given the same separability by construction, on a dataset that splits roughly 61, 25, 9 and 4 percent. The AUROCs are not the same: 0.8517, 0.8494, 0.8602, 0.8701. The rarest class scores highest.

The generator treats the classes symmetrically apart from their sample counts, so this run does not isolate a population effect of negative-mixture composition. Finite-sample variation also contributes to the displayed differences, especially for the smaller classes. In other data, changing the mixture can change OvR AUROC when the model’s score distributions differ across negative classes. Each value describes a particular evaluation population.

Two consequences follow, and both are commonly ignored:

• Do not rank the classes by their OvR AUROC. “The model is best at class 3” does not follow from 0.8701 being the largest, because no two of these numbers are measured against the same thing.
• The values move when the class mix moves. Fawcett makes the point directly — the class-reference construction “compromises one of the attractions of ROC graphs, namely that they are insensitive to class skew”. Changing how much of the data some other class occupies changes this class’s number, which is measured and quantified under macro AUROC.

Where the classes genuinely need comparing against each other, one-versus-one is the construction that does it — see multiclass AUROC.

What it is good for

One-versus-rest matches how most systems are actually used. A fraud model answers “is this fraud?”, not “is this fraud rather than a chargeback?”, and the OvR AUROC for the fraud class is the quantity that corresponds to that question.

It is also stable in one respect that matters on imbalanced data: AUROC does not move with the positive class’s own prevalence, so a class holding 4 percent of the data gets a number on the same scale as one holding 61 percent. That is a real advantage over accuracy, and it is not a claim about usefulness in production — a high OvR AUROC on a rare class is compatible with an alert queue that is almost entirely false alarms, because the false positive rate divides by a very large negative set. Saito and Rehmsmeier attribute that trap to “an intuitive but wrong interpretation of specificity”; pair the value with a precision-side figure at the operating point, as under PR-AUC.

Report the per-class values with their supports and, where the reader might compare them, a note on what each “rest” contains.

References

• Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8), 2006, 861–874
• Takaya Saito and Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 10(3), 2015
• scikit-learn API reference, roc_auc_score