Macro AUC

Macro AUC computes an area per class, one-versus-rest, and averages the results with equal weight:

\(\text{AUC}_{\text{macro}} = \dfrac{1}{K}\sum_{i=1}^{K}\text{AUC}_i\)

Per-class values of 0.95, 0.80 and 0.60 give 0.78, and a class holding 1 percent of the data counts as much as one holding half. That is the whole idea, and it is not where the difficulty is.

Which area

The formula never says what \(\text{AUC}_i\) is. It is usually the area under a ROC curve, and it is sometimes the area under a precision–recall curve. Those are different numbers with different no-skill baselines — 0.5 for one, the class’s prevalence for the other — and averaging them across classes does not bring them closer.

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

rng = np.random.default_rng(6)
names = ["common", "medium", "rare"]
support = dict(zip(names, (4000, 800, 120)))
y = np.concatenate([[k] * n for k, n in enumerate(support.values())])
onehot = np.eye(len(names))[y]
scores = rng.normal(size=onehot.shape) + 1.2 * onehot
probs = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)

per_roc = [roc_auc_score(onehot[:, k], probs[:, k]) for k in range(len(names))]
per_pr = [average_precision_score(onehot[:, k], probs[:, k]) for k in range(len(names))]
for name, roc, pr, n in zip(names, per_roc, per_pr, support.values()):
    print(f"{name:7} ROC-AUC {roc:.4f}  AP {pr:.4f}  prevalence {n / len(y):.4f}")
print(f"macro   ROC-AUC {np.mean(per_roc):.4f}  AP {np.mean(per_pr):.4f}")
# common  ROC-AUC 0.8668  AP 0.9619  prevalence 0.8130
# medium  ROC-AUC 0.8616  AP 0.5907  prevalence 0.1626
# rare    ROC-AUC 0.8518  AP 0.2335  prevalence 0.0244
# macro   ROC-AUC 0.8601  AP 0.5954

The classes have similar ROC-AUCs, 0.8668, 0.8616 and 0.8518, but average precision (AP) is 0.9619, 0.5907 and 0.2335. The code uses AP, a recall-increment-weighted summary of precision; it does not compute a trapezoidal PR-curve area. Prevalence contributes to the difference between the class APs, so report it alongside each result rather than treating the two kinds of area as interchangeable.

So “macro AUC = 0.86” and “macro AUC = 0.60” here describe the same predictions. Report which curve, and if it is the PR side, report each class’s prevalence alongside — a macro AP of 0.60 against baselines of 0.81, 0.16 and 0.02 is not interpretable without them.

The other averages

“Weighted macro AUC” appears in practitioner writing and is worth avoiding: weighting by support is the thing macro averaging exists to refuse, so the phrase names a scheme that is the opposite of what it says.

Before quoting one

A macro AUC is complete when it states the curve, the multi-class construction, and the per-class values it averages. The construction matters more than it looks: one-versus-rest gives up ROC’s insensitivity to class balance, while one-versus-one does not, and the two can differ by several points on the same data — that is set out under macro AUROC, which is the same metric under its unambiguous name.

Where the word “AUC” is doing the same work in a binary setting, the same question applies and is covered under AUC.

References

• scikit-learn API reference, roc_auc_score
• scikit-learn API reference, average_precision_score
• Takaya Saito and Marc Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 10(3), 2015