Macro Recall

Macro recall computes recall for each class one-versus-rest and averages the results with equal weight:

\(\text{Recall}_{\text{macro}} = \dfrac{1}{K}\sum_{i=1}^{K}\text{Recall}_i\)

Per-class recalls of 0.90, 0.60 and 0.30 give 0.60, and the third class counts as much as the first however few samples it has. That is the whole design: it asks how well the model does on a typical class rather than a typical sample.

The same number under another name

Balanced accuracy is defined as the mean of the per-class recalls. That is the definition above, so the two are the same quantity — on the classes that are actually present. If the evaluation label set names a class with no true members, its recall is 0/0, macro recall falls back on zero_division, and the two can part company. Fix the label set to the classes present, or state the rule.

import numpy as np
from sklearn.metrics import recall_score, precision_score, f1_score, balanced_accuracy_score

rng = np.random.default_rng(6)
names = ["common", "medium", "rare"]
support = dict(zip(names, (4000, 800, 120)))
hit_rate = dict(zip(names, (0.95, 0.70, 0.35)))
truth, pred = [], []
for name, n in support.items():
    truth += [name] * n
    others = [k for k in names if k != name]
    pred += [name if rng.uniform() < hit_rate[name] else rng.choice(others) for _ in range(n)]

macro_r = recall_score(truth, pred, average="macro", labels=names, zero_division=0)
macro_p = precision_score(truth, pred, average="macro", labels=names, zero_division=0)
print(f"macro recall        {macro_r:.6f}")
print(f"balanced accuracy   {balanced_accuracy_score(truth, pred):.6f}")
print(f"macro precision     {macro_p:.6f}")
print(f"macro F1            {f1_score(truth, pred, average='macro', labels=names, zero_division=0):.6f}")
print(f"harmonic mean of the two macro numbers {2 * macro_p * macro_r / (macro_p + macro_r):.6f}")
# macro recall        0.683278
# balanced accuracy   0.683278
# macro precision     0.650434
# macro F1            0.650673
# harmonic mean of the two macro numbers 0.666452

0.683278 both ways, to six decimal places. This is worth knowing in both directions. If a paper reports balanced accuracy and you want macro recall, you already have it. And balanced accuracy comes with a reading that macro recall is usually quoted without: a model guessing uniformly at random scores \(1/K\) in expectation, so 0.68 on a three-class problem should be read against 0.33 rather than against 0. \(1/K\) is a reference point, not a floor: the minimum is 0, reached by a model that gets every class wrong.

That reference point is what makes macro recall the useful counterpart to accuracy on imbalanced data. Plain accuracy rewards a model for ignoring rare classes. Macro recall does not: a model that puts everything in one class scores \(1/K\) whatever the class mix, because the ignored classes each contribute zero. A model that ignores one class out of three and is imperfect on the others scores below \(2/3\), not \(1/K\) — the \(1/K\) figure belongs specifically to predicting a single class throughout.

Macro F1 is not the F1 of the macros

The last two lines of the output are the same model’s macro F1 and the harmonic mean of its macro precision and macro recall: 0.650673 against 0.666452.

They are different calculations. Macro F1 computes F1 within each class and averages the results; the harmonic mean of the two macro averages averages first and combines second. Averaging and the harmonic mean do not commute, so the two differ whenever precision and recall vary unevenly across classes — which is the situation macro averaging exists for.

The gap here is 0.016, small enough to look like rounding and large enough to change a comparison between two models. Report the macro F1 you computed rather than one derived from the other two numbers in the table.

Steadier than macro precision

Macro recall and macro precision look symmetric and are not, because their denominators come from different places.

So macro recall degrades gracefully on the failure it is meant to detect — a class the model never predicts simply scores 0 and drags the average down, which is the right answer. Macro precision on the same predictions produces 0.27, 0.81 or 0.94 depending on a parameter.

That makes macro recall the more robust single number for imbalanced multiclass work, and it is still not sufficient on its own: a model that predicts every class liberally has high recall everywhere and useless precision. Report it with macro precision, or with the per-class table underneath both — see per-class precision.

References

• scikit-learn API reference, recall_score
• scikit-learn API reference, balanced_accuracy_score