Multiclass Classification

Multiclass classification assigns each input exactly one label from three or more options — a handwritten digit, an animal species, a support-ticket category. Exactly one: if a sample can carry several labels at once, that is multi-label classification and a different problem.

Formally, learn \(f: X \to Y\) where \(Y = \{1, 2, \dots, K\}\) and \(K > 2\). Three routes get you there:

• Softmax — one model producing a distribution over all \(K\) classes. This is what scikit-learn’s LogisticRegression does today; its documentation states that for three or more classes every solver except liblinear optimizes the multinomial loss, and the old multi_class switch is gone.
• One-vs-Rest — \(K\) binary models, one per class, highest score wins.
• One-vs-One — \(K(K-1)/2\) models, one per pair, decided by majority vote. Standard in SVM implementations and quadratic in \(K\).

The harder question is not which route to take. It is how to tell whether the result is any good.

A healthy accuracy with one class missing

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=10, n_informative=7, n_classes=5,
                           n_clusters_per_class=1, weights=[0.45, 0.30, 0.15, 0.08, 0.02],
                           random_state=0)
xtr, xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
model = LogisticRegression(max_iter=5000).fit(xtr, ytr)
pred = model.predict(xte)

print(f"overall accuracy {accuracy_score(yte, pred):.4f}"
      f"   micro F1 {f1_score(yte, pred, average='micro'):.4f}"
      f"   macro F1 {f1_score(yte, pred, average='macro'):.4f}\n")
per_class = recall_score(yte, pred, average=None)
for k in range(5):
    print(f"  class {k}  share of the data {np.mean(yte == k):.4f}"
          f"   recall {per_class[k]:.4f}   times predicted {np.mean(pred == k):.4f}")
# overall accuracy 0.8575   micro F1 0.8575   macro F1 0.6922
#
#   class 0  share of the data 0.4462   recall 0.9178   times predicted 0.4707
#   class 1  share of the data 0.2997   recall 0.8537   times predicted 0.3098
#   class 2  share of the data 0.1512   recall 0.8908   times predicted 0.1513
#   class 3  share of the data 0.0812   recall 0.6982   times predicted 0.0667
#   class 4  share of the data 0.0218   recall 0.0382   times predicted 0.0015

Accuracy 0.8575. Micro F1 0.8575. Both defensible-looking numbers, and class 4 has a recall of 0.0382 — it is 2.18% of the data and the model names it for 0.15% of rows. In practice the class does not exist as far as this model is concerned.

Macro F1 is 0.6922, sixteen points below micro, and that gap is the signal. It appears because the two averages ask different questions.

• Micro averaging pools every prediction and then computes the metric, so each sample gets one vote. Large classes dominate.
• Macro averaging computes the metric per class and then averages, so each class gets one vote regardless of size. A class the model ignores drags it down.

Neither is correct in general. If misclassifying a rare disease matters as much as misclassifying a common one, macro is the right summary. If every sample carries equal cost, micro is. The mistake is reporting only one and treating it as “the” F1.

One identity is worth knowing because it causes confusion: in single-label multiclass, micro F1 equals accuracy exactly — both are 0.8575 above. Every prediction is either right or wrong, so a false positive for one class is a false negative for another and the pooled counts collapse. Quoting accuracy and micro F1 side by side gives two numbers and one fact.

What to look at instead

• The confusion matrix, always. Not a summary of it — the matrix. It shows which classes are being confused with which, which no scalar can.
• Per-class recall next to class frequency. The two columns together are what revealed class 4 above.
• How often each class is predicted. A class that is never predicted has been dropped, and per-class precision can stay undefined or misleadingly high while that happens.
• Report macro and micro with the per-class results and supports. Their difference can reveal a consequence of aggregation, but it does not measure class disparity on its own. With two equally supported classes whose recalls are 1.0 and 0.5, macro and micro recall are both 0.75 despite the unequal performance. For pooled AUC, cross-class score scales add another source of differences.

If a rare class matters, the remedies are the usual ones and they cost the same as always — a class weight, a per-class threshold, or more data for that class. As the article on class weighting shows, buying recall on the rare class costs precision elsewhere and distorts the predicted probabilities, so the decision belongs to whoever owns the consequence.

And scale matters for the strategy: at a thousand classes, One-vs-One’s half-million models are out of the question, One-vs-Rest gives a thousand badly imbalanced binary problems, and a single softmax is both cheaper and better posed.

References

• scikit-learn API reference, LogisticRegression
• scikit-learn User Guide, Metrics and scoring
• scikit-learn User Guide, Multiclass and multioutput algorithms