Cross-Validation (CV)

Cross-validation splits the data into \(k\) folds, trains on \(k-1\) of them and tests on the one left out, rotating until every fold has served as the test set once, then averages the \(k\) scores. A single train/test split gives one number that depends on which rows happened to land where; cross-validation averages that away.

What the number refers to

Cross-validation trains \(k\) different models and never keeps any of them. What gets deployed is refit on all the data, and that model has no cross-validated score of its own.

So the estimate belongs to the procedure: this algorithm, with these hyperparameters and this preprocessing, applied to a dataset of about this size. That is usually what you want when comparing two approaches, and it is not the same as an estimate of the thing you shipped. It is also slightly pessimistic for that thing, since each fold’s model trained on less data than the final one.

How much the number moves

import numpy as np
from sklearn.model_selection import cross_val_score, KFold
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(1)
model = DecisionTreeClassifier(random_state=0)

def one_dataset(n=200, seed=7):
    r = np.random.default_rng(seed)
    x = r.normal(size=(n, 4))
    return x, (r.uniform(size=n) < 1 / (1 + np.exp(-x[:, 0]))).astype(int)

x, y = one_dataset()
across_seeds = [cross_val_score(model, x, y, cv=KFold(5, shuffle=True, random_state=s)).mean()
                for s in range(50)]
print(f"one dataset, 5-fold over 50 split seeds  min {min(across_seeds):.4f}  "
      f"max {max(across_seeds):.4f}  sd {np.std(across_seeds, ddof=1):.4f}")

means, naive = [], []
for _ in range(150):
    xx = rng.normal(size=(200, 4))
    yy = (rng.uniform(size=200) < 1 / (1 + np.exp(-xx[:, 0]))).astype(int)
    scores = cross_val_score(model, xx, yy, cv=KFold(10, shuffle=True, random_state=0))
    means.append(scores.mean())
    naive.append(scores.std(ddof=1) / np.sqrt(10))
true_sd = float(np.std(means, ddof=1))
print(f"10-fold: true sd of the estimate {true_sd:.4f}  "
      f"naive SE from the folds {np.mean(naive):.4f}  ratio {true_sd / np.mean(naive):.2f}")
# one dataset, 5-fold over 50 split seeds  min 0.4700  max 0.6150  sd 0.0330
# 10-fold: true sd of the estimate 0.0476  naive SE from the folds 0.0341  ratio 1.40

The first line takes one dataset and one model and varies only the shuffling seed. Five-fold cross-validation returns anything from 0.4700 to 0.6150. Nothing about the data or the model differs between those runs.

The second line addresses the error bar usually quoted alongside — the standard deviation across folds divided by \(\sqrt{k}\). Measured against the estimate’s true spread over fresh datasets, that figure is too small by a factor of 1.40 here. The reason is that the folds are not independent: any two training sets share most of their rows, so their errors are correlated and the usual formula for the standard error of a mean does not apply.

Bengio and Grandvalet show why a universal unbiased variance estimator for K-fold CV is unavailable without further assumptions. That does not establish a universal one-percentage-point significance threshold. Compare models using their paired performance differences, an uncertainty procedure appropriate to the sampling dependence, and the scale that matters for the application.

Which variant

The last row is the one most often skipped. Choosing a model by its cross-validated score and then reporting that score describes the best of many draws, not the procedure — the same reason a maximum over noisy estimates is biased upward.

What it cannot rescue

Cross-validation protects against one thing: a lucky or unlucky single split. It does nothing about a preprocessing step fitted on all the data before the folds were drawn, which leaks the test set into training and can turn pure noise into an apparently strong model — that is measured under k-fold cross-validation. Nor does it help when the evaluation data does not look like production data, which no resampling scheme can detect.

References

• Yoshua Bengio and Yves Grandvalet, “No Unbiased Estimator of the Variance of K-Fold Cross-Validation”, Journal of Machine Learning Research, 5, 2004, 1089–1105
• scikit-learn User Guide, Cross-validation: evaluating estimator performance