Macro Precision

Macro precision takes the per-class precision values and averages them with equal weight:

\(\text{Precision}_{\text{macro}} = \dfrac{1}{K}\sum_{i=1}^{K}\text{Precision}_i\)

Each class is treated one-versus-rest — that class is positive, all others are negative — and every class contributes \(1/K\) regardless of how many samples it has. With per-class precisions of 0.80, 0.60 and 0.40, macro precision is 0.60, and it stays 0.60 whether the third class has 4,000 members or 12.

scikit-learn documents the intent plainly: the macro average calculates the metric for each label and finds “their unweighted mean”, which “does not take label imbalance into account”. That is the feature. It is also what makes the number unstable in exactly the situation it exists for.

The undefined class

Precision for class \(c\) divides by \(TP_c + FP_c\), the number of times the model predicted \(c\). If it never predicts \(c\), that is zero and the precision is undefined. A model ignoring a rare class is the standard failure mode on imbalanced data, so this is not a corner case.

import numpy as np
from sklearn.metrics import precision_score

names = ["common", "medium", "rare"]
truth = ["common"] * 4000 + ["medium"] * 800 + ["rare"] * 120
timid = ["common"] * len(truth)
"""The timid model never predicts anything but the majority class, so two of the
   three per-class precisions are 0/0 and the macro average is whatever you decide
   0/0 should be."""

for zero_division in (0, 1, np.nan):
    per_class = precision_score(truth, timid, average=None, labels=names, zero_division=zero_division)
    macro = precision_score(truth, timid, average="macro", labels=names, zero_division=zero_division)
    print(f"zero_division={str(zero_division):5} per-class {np.array2string(per_class, precision=4)}  macro {macro:.4f}")

rng = np.random.default_rng(6)
hit_rate = dict(zip(names, (0.95, 0.70, 0.35)))
support = dict(zip(names, (4000, 800, 120)))
pred = []
for name, n in support.items():
    others = [k for k in names if k != name]
    pred += [name if rng.uniform() < hit_rate[name] else rng.choice(others) for _ in range(n)]
for scheme in ("macro", "micro", "weighted"):
    print(f"a real model: {scheme:8} precision "
          f"{precision_score(truth, pred, average=scheme, labels=names, zero_division=0):.4f}")
# zero_division=0     per-class [0.813 0.    0.   ]  macro 0.2710
# zero_division=1     per-class [0.813 1.    1.   ]  macro 0.9377
# zero_division=nan   per-class [0.813   nan   nan]  macro 0.8130
# a real model: macro    precision 0.6504
# a real model: micro    precision 0.8982
# a real model: weighted precision 0.9182

One model, one set of predictions, three macro precisions: 0.2710, 0.9377 and 0.8130. Nothing about the model differs between the rows. What differs is the answer to “what is 0/0”, and scikit-learn lets you choose 0, 1 or nan.

Set it explicitly. A macro precision reported without saying which convention produced it is not reproducible, and the three answers here span 0.67 of the metric’s range.

Against the other averages

The last three lines take a model that does predict every class, with per-class recalls around 0.95, 0.70 and 0.35 on supports of 4,000, 800 and 120.

A quarter of a point separates macro from micro on the same predictions, and neither is wrong. Macro is the right choice when a rare class matters as much as a common one — a disease that affects few people, a fraud pattern that is rare and expensive, a language with little training data. Micro is right when the cost is per sample and every sample counts the same.

Weighted is the one most likely to be reported without thought, because it is closest to what a naive reader expects a single number to mean. It is not guaranteed to fall between the other two — here it is 0.9182, above micro’s 0.8982 — because it weights the per-class values by true support while micro weights them by predicted count. It is also the least informative of the three: it tracks the common classes almost as closely as micro while carrying the per-class machinery of macro.

Two further cautions

Macro precision has high variance when a class is small. A class with 12 test samples contributes a full \(1/K\) of the score and its precision is estimated from a handful of predictions, so macro moves noticeably between runs for reasons that have nothing to do with the model.

And macro F1 is not the F1 of macro precision and macro recall. It is the mean of the per-class F1 values, and averaging before and after the harmonic mean gives different answers — measured at 0.6507 against 0.6665 on the model above. Report the one you computed, not the one that would follow from the other two.

References

• scikit-learn API reference, precision_score
• scikit-learn User Guide, Metrics and scoring