Multiclass stratified CV

Multiclass stratified cross-validation extends stratification past two classes: every fold is drawn to hold roughly the same proportion of each class as the full dataset.

With 1,000 samples split 600 / 300 / 100 and five folds, each fold gets about 120, 60 and 20. Ordinary k-fold averages to the same thing and varies around it, which is how a fold ends up with three samples of the smallest class and another with thirty.

Where the guarantee runs out

Stratification is usually described as guaranteeing that every fold represents every class. It cannot, and the boundary is arithmetic: a class with fewer members than there are folds has nothing to spread.

import numpy as np
import warnings
from sklearn.model_selection import StratifiedKFold

rng = np.random.default_rng(5)
supports = [400, 200, 60, 12, 3]
y = np.concatenate([[k] * n for k, n in enumerate(supports)])
x = np.zeros((y.size, 1))

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    folds = list(StratifiedKFold(5, shuffle=True, random_state=1).split(x, y))
    messages = [str(w.message).split(".")[0] for w in caught]
table = np.array([[int((y[test] == k).sum()) for k in range(len(supports))] for _, test in folds])
for i, row in enumerate(table):
    print(f"fold {i}  per-class test counts {row.tolist()}")
print(f"column sums {table.sum(axis=0).tolist()}  matches supports {supports}")
print(f"warnings: {messages[0] if messages else 'none'}")
# fold 0  per-class test counts [80, 40, 12, 3, 0]
# fold 1  per-class test counts [80, 40, 12, 3, 0]
# fold 2  per-class test counts [80, 40, 12, 2, 1]
# fold 3  per-class test counts [80, 40, 12, 2, 1]
# fold 4  per-class test counts [80, 40, 12, 2, 1]
# column sums [400, 200, 60, 12, 3]  matches supports [400, 200, 60, 12, 3]
# warnings: The least populated class in y has only 3 members, which is less than n_splits=5

Five classes with 400, 200, 60, 12 and 3 members. The first three are divided exactly — 80, 40 and 12 per fold, five times over. The fourth splits 3, 3, 2, 2, 2, which is as even as twelve across five folds gets. The fifth has three members and reaches three folds; two folds contain none of it.

scikit-learn does not refuse. It emits a warning — “The least populated class in y has only 3 members, which is less than n_splits=5” — and proceeds, so a run that logs warnings to a file produces cross-validated scores that look ordinary and were computed on folds where one class did not exist.

Two things follow from that. Stratification across many classes is approximate, not exact, unless every class count divides the fold count. And the useful check is not whether stratification was requested but whether the smallest class has at least as many members as there are folds, with enough left over that each fold’s share is worth estimating from.

What the empty folds do to the score

A fold with no members of class \(c\) has no recall for \(c\) — it is \(0/0\). Under macro averaging that becomes whatever zero_division says, and the fold’s macro score then depends on a convention rather than on the model. The fold-level numbers still average into a headline figure that carries no sign of it.

Fewer folds is usually the better response than more stratification. Three folds on the example above would give the three-member class one sample each, which is thin but defined. The alternatives are to merge the smallest classes into an “other” category, or to accept that this class cannot be evaluated by cross-validation on this dataset and say so.

When to use it

• By default for multiclass classification. It costs nothing and removes a source of fold-to-fold noise. scikit-learn applies it automatically when cross_val_score receives a classifier.
• Especially when reporting macro-averaged metrics, since those weight every class equally and are the most exposed to a fold missing one — see macro recall.
• Not on its own when rows are grouped. Keeping class proportions and keeping a user’s rows together are separate constraints; stratified group k-fold does both, at the cost of satisfying each less exactly.

Where repeated random train/test pairs are wanted instead of a partition, the stratified equivalent is stratified shuffle split.

References

• scikit-learn API reference, StratifiedKFold
• scikit-learn User Guide, Cross-validation: evaluating estimator performance