Single-label Classification

Single-label classification assigns each sample exactly one class from \(Y = \{1, 2, \dots, K\}\). It is the umbrella term covering binary classification when \(K = 2\) and multiclass when \(K > 2\), and it is what people usually mean by “classification” without a qualifier.

The decision rule is a maximum:

\[ f(x) = \arg\max_{k} P(y = k \mid x) \]

The defining property is that the classes are mutually exclusive — a sample cannot be two of them. It is worth being precise about where that property comes from, because it is usually described as a fact about the data and it is really a constraint on the model.

What exclusivity costs when it is not true

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.multioutput import MultiOutputClassifier

rng = np.random.default_rng(0)
n, K, d = 20000, 8, 12
Z = rng.normal(size=(n, d))
theme = rng.normal(size=(2, d))
loadings = np.abs(rng.normal(size=(K, 2))) + 0.3
Y = (rng.random((n, K)) < 1 / (1 + np.exp(-((Z @ theme.T) @ loadings.T - 1.0)))).astype(int)
"""The world here is multi-label: a row can carry several tags at once."""

counts = Y.sum(axis=1)
print("how many tags each row really carries")
for k in range(5):
    print(f"  {k} tags: {np.mean(counts == k):.4f}")
print(f"  {5} or more: {np.mean(counts >= 5):.4f}")
print(f"rows where a single answer cannot be complete: {np.mean(counts >= 2):.4f}\n")

keep = counts >= 1
first = np.argmax(Y[keep], axis=1)
"""Forcing one label per row: keep the first tag, discard the rest, fit a softmax."""
xtr, xte, ytr, yte = train_test_split(Z[keep], first, test_size=0.3, random_state=0)
forced = LogisticRegression(max_iter=5000).fit(xtr, ytr)
_, full_te = train_test_split(Y[keep], test_size=0.3, random_state=0)
pred = forced.predict(xte)
hit = full_te[np.arange(len(pred)), pred] == 1
print(f"the forced model reproduces its own single target      {np.mean(pred == yte):.4f}")
print(f"its one answer is among the row's real tags            {hit.mean():.4f}")
print(f"average real tags per row (of those kept)              {full_te.sum(axis=1).mean():.4f}")
print(f"share of the real tags one answer can recover          {hit.sum() / full_te.sum():.4f}")
# how many tags each row really carries
#   0 tags: 0.2984
#   1 tags: 0.1278
#   2 tags: 0.0891
#   3 tags: 0.0732
#   4 tags: 0.0691
#   5 or more: 0.3425
# rows where a single answer cannot be complete: 0.5738
#
# the forced model reproduces its own single target      0.5895
# its one answer is among the row's real tags            0.6093
# average real tags per row (of those kept)              4.4580
# share of the real tags one answer can recover          0.1367

This data is genuinely multi-label: rows carry several tags. Forcing a single answer onto it — keeping one tag per row and fitting a softmax — produces a model that works, in the sense that it reproduces its own target 0.5895 of the time.

The problem is what the target became. 57.38% of rows carry two or more tags, so for the majority of the data no single answer can be complete. The model’s one guess lands inside the true tag set 0.6093 of the time, and across the dataset it recovers 13.67% of the tags that are actually there.

None of that shows up in the model’s own accuracy figure, because the evaluation inherited the same constraint as the training. The additional tags were discarded from the training target. Metrics computed only against that reduced target cannot reveal the omission; evaluation against the original tag set can.

(The construction here is synthetic, and 29.84% of its rows carry no tag at all. The point is the structure — most rows needing more than one answer — rather than the exact proportions.)

The diagnostic is a question about the labeling process, asked before the model is chosen: could an annotator have legitimately ticked two boxes? Medical diagnosis is the standard example, and the standard example is also where the assumption most often fails — comorbidity is common, and a patient with diabetes and hypertension has two conditions, not one that beat the other.

A single-label model must always answer

A softmax distributes probability across the classes and it sums to one, so there is no way to express “none of these.” The model will name the least implausible class for an input unlike anything it has seen.

Two usual repairs. Add an explicit “other” class, which requires examples of it and is only as good as that collection. Or threshold the maximum probability and abstain below it, which is a rejection option rather than a classification — and it needs calibrated probabilities to be set sensibly.

The argmax is optimal only under equal costs

Picking the highest-probability class minimizes the expected number of mistakes — the 0-1 loss. That is the right objective when every error costs the same.

Where it does not, the argmax is the wrong rule. If mistaking a malignant tumor for benign costs fifty times the reverse, the class to predict is the one minimizing expected cost, and it can be a class with 10% probability rather than the one with 60%. The general form weights each class’s probability by the cost of choosing it wrongly — the same reasoning that puts the threshold away from 0.5 in the two-class case.

So single-label classification is the simplest setting and it still has two decisions in it that the model does not make: whether one answer is the right shape for the problem, and what the answers cost. Where the first is no, multi-label classification is the setting — with its own consequences for metrics and thresholds.

References

• scikit-learn User Guide, Multiclass and multioutput algorithms
• scikit-learn User Guide, Metrics and scoring