Multiclass Precision

In a multiclass problem each sample belongs to exactly one of several classes — a digit is a 3 or a 7, not both. Precision is still defined one class at a time, treating that class as positive and the others as negative:

\(\text{Precision}_c = \dfrac{TP_c}{TP_c + FP_c}\)

which answers, for each class in turn, how often the model is right when it names that class. The result is a list. Turning it into one number is a separate decision, and the three standard ways of doing it give genuinely different answers.

import numpy as np
from sklearn.metrics import precision_score, accuracy_score, confusion_matrix

rng = np.random.default_rng(11)
names = ["digit 0", "digit 1", "digit 2"]
support = dict(zip(names, (3000, 600, 90)))
hit_rate = dict(zip(names, (0.96, 0.72, 0.30)))
truth, pred = [], []
for name, n in support.items():
    truth += [name] * n
    others = [k for k in names if k != name]
    pred += [name if rng.uniform() < hit_rate[name] else rng.choice(others) for _ in range(n)]

per_class = precision_score(truth, pred, average=None, labels=names, zero_division=0)
predicted = confusion_matrix(truth, pred, labels=names).sum(axis=0)
for name, p, k in zip(names, per_class, predicted):
    print(f"{name:8} precision {p:.4f}  support {support[name]:5d}  predicted {k:5d}")
for scheme in ("macro", "weighted", "micro"):
    print(f"{scheme:9} {precision_score(truth, pred, average=scheme, labels=names, zero_division=0):.6f}")
print(f"accuracy  {accuracy_score(truth, pred):.6f}")
print(f"micro rebuilt as a predicted-count weighted mean  "
      f"{float(np.average(per_class, weights=predicted)):.6f}")
# digit 0  precision 0.9656  support  3000  predicted  2990
# digit 1  precision 0.8204  support   600  predicted   540
# digit 2  precision 0.1688  support    90  predicted   160
# macro     0.651557
# weighted  0.922511
# micro     0.909756
# accuracy  0.909756
# micro rebuilt as a predicted-count weighted mean  0.909756

Three classes holding 3,000, 600 and 90 samples. The per-class precisions are 0.9656, 0.8204 and 0.1688, and the three averages are 0.6516, 0.9225 and 0.9098 — a spread of a quarter of the scale on one set of predictions.

What each average weights by

The last two rows are worth separating carefully, because they are usually described as if they were the same idea. Micro precision pools \(TP\) and \(FP\) before dividing, and the last line of the output shows that this is exactly the per-class values averaged by predicted count — 0.909756 either way. Weighted precision averages the same values by true support. Different weights, different numbers, and no guarantee that weighted falls between macro and micro: here it sits above both.

Micro’s other identity is the more consequential one. In a single-label problem every mistake is a false positive for one class and a false negative for another, so the pooled counts collapse and micro precision equals accuracy exactly. If accuracy was set aside because the classes are imbalanced, micro precision does not address that.

The class the model never predicts

Precision divides by the number of times the class was predicted, so a class the model ignores gives \(0/0\). On imbalanced data that is the standard failure, not an edge case, and scikit-learn’s zero_division decides what appears — 0, 1 or nan. The choice moves macro precision a long way; the sizes are measured under macro precision. Set it explicitly.

Reporting it

A multiclass precision is complete with the averaging scheme, the zero_division convention, and the per-class table with supports and predicted counts underneath — comparing those last two columns is what distinguishes a class the model cannot detect from one it over-predicts, and they call for different fixes. Report recall beside it, since precision alone is raised by predicting the class less often.

Where a sample can carry several labels at once, the accounting changes and so do the averages available — see multilabel precision.

References

• scikit-learn API reference, precision_score
• scikit-learn User Guide, Metrics and scoring