Macro F1

Macro F1 computes an F1 score for each class one-versus-rest and averages them with equal weight:

\(F1_{\text{macro}} = \dfrac{1}{K}\sum_{i=1}^{K} F1_i\)

With per-class precision and recall of (0.80, 0.80), (0.60, 0.40) and (0.50, 0.25), the class F1 scores are 0.80, 0.48 and 0.33, and macro F1 is 0.54. One class doing well does not rescue the average, which is the property the scheme exists for: a rare class counts as much as a common one.

It is not the F1 of the macro averages

There are two ways to combine per-class precision and recall into one number, and they are not the same calculation. Macro F1 takes the harmonic mean within each class and then averages. The alternative — averaging precision and recall first, then taking their harmonic mean — is a different quantity that is easy to write by accident when a report lists macro precision and macro recall next to each other.

import numpy as np
from sklearn.metrics import f1_score, precision_score, recall_score

rng = np.random.default_rng(6)
names = ["common", "medium", "rare"]
support = dict(zip(names, (4000, 800, 120)))
hit_rate = dict(zip(names, (0.95, 0.70, 0.35)))
truth, pred = [], []
for name, n in support.items():
    truth += [name] * n
    others = [k for k in names if k != name]
    pred += [name if rng.uniform() < hit_rate[name] else rng.choice(others) for _ in range(n)]

kw = dict(labels=names, zero_division=0)
macro_p = precision_score(truth, pred, average="macro", **kw)
macro_r = recall_score(truth, pred, average="macro", **kw)
print(f"macro F1                     {f1_score(truth, pred, average='macro', **kw):.6f}")
print(f"F1 of macro P and macro R    {2 * macro_p * macro_r / (macro_p + macro_r):.6f}")
print(f"per-class F1                 " +
      " ".join(f"{v:.4f}" for v in f1_score(truth, pred, average=None, **kw)))

timid = ["common"] * len(truth)
"""A model that only predicts the majority class leaves precision undefined for the other two classes; their count-based F1 is zero."""
for zero_division in (0, 1):
    print(f"timid model, zero_division={zero_division}  macro F1 "
          f"{f1_score(truth, timid, average='macro', labels=names, zero_division=zero_division):.4f}")
# macro F1                     0.650673
# F1 of macro P and macro R    0.666452
# per-class F1                 0.9588 0.7419 0.2513
# timid model, zero_division=0  macro F1 0.2990
# timid model, zero_division=1  macro F1 0.2990

0.650673 against 0.666452 on the same predictions. The harmonic mean is not linear, so it does not commute with averaging, and the gap widens as precision and recall diverge unevenly across classes — which is the situation macro averaging is used in. Report the one you computed; do not derive it from the other two numbers in the table.

When F1 is undefined

Macro precision has an awkward failure on imbalanced data: a class the model never predicts has precision \(0/0\), and the zero_division setting moves macro precision between 0.2710 and 0.9377 on one set of predictions.

If a class has true instances but is never predicted, its count-based F1 is 0: the denominator in 2TP/(2TP+FP+FN) is positive. This is why the example gives 0.2990 under both settings even though precision needs a convention. F1 itself is undefined when TP, FP and FN are all zero, as can happen when an explicitly included label is absent from both truth and prediction. Including such a class lets zero_division change macro F1, so report the label set and convention.

Reading it with the per-class values

The per-class values are 0.9588, 0.7419 and 0.2513 on supports of 4,000, 800 and 120. Their average is 0.6507. Each class receives one third of the weight, including the rare class whose score is estimated from only 120 samples; report the supports to make that unequal precision visible.

Use macro F1 where every class matters regardless of size — a disease that is rare and serious, a fraud pattern that is rare and expensive, a language with little data. Report the per-class table underneath it, because the average is the thing that hides what you were trying to see.

Absent classes also affect the conventions used for macro recall.

References

• scikit-learn API reference, f1_score
• scikit-learn User Guide, Metrics and scoring