Train, Validation, and Test Sets without Data Leakage

Leakage occurs when model development uses information that would be unavailable for the intended prediction, or when data reserved for independent evaluation influence fitting or selection. It can make a model look useful for a task it has not actually been tested on. The examples below examine feature selection, repeated observations from the same subject, and forecasting with time-ordered data.

The rule

Training data are used to fit the model and its preprocessing. Validation data help choose settings or compare approaches. Test data provide a final evaluation after those choices are settled. A validation score used to choose the winner should not also be presented as an independent assessment of that winner. Reserve the test set before development, and choose a split that matches who or what the model will predict for.

For a model chosen independently of an i.i.d. evaluation sample from the target distribution, the average evaluation loss is an unbiased estimate of its risk, provided the expected loss is finite. Using evaluation data during fitting or selection removes that guarantee. The distortion depends on how the information affects the fitted model and its evaluation; it need not increase the score in every run. The empirical risk and generalization article develops this distinction.

Within each training split, fit the feature selector, scaler, or encoder using only that split, then apply the fitted transformation to its held-out portion. Validation can choose the preprocessing method or its settings. A pipeline bundles these fitted steps with the model so cross-validation refits the whole procedure on each training portion. A transformation fitted on the entire dataset before cross-validation has already used its held-out rows. Fixed operations, such as converting centimeters to meters, do not estimate anything from the data and can be applied before splitting. For fitted transformations, including imputation and scaling, the training boundary matters.

Leak one: selecting features on all the data

A label-based feature screen can pick chance associations if it sees evaluation labels. Here SelectKBest uses f_classif to rank features by differences between the two classes and keeps 20. In five-fold cross-validation, each fold is a held-out group of 40 of the 200 rows. The pipeline selects features using the other 160 rows, fits the classifier there, and applies the same selected columns to the 40 held-out rows. It repeats this with each fold held out once. For this binary classifier, cv=5 uses stratified folds, keeping class proportions approximately similar, and cross_val_score reports accuracy by default. The printed value is the mean of the five fold accuracies. Stratification by itself does not keep subjects together or preserve time order.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
n, p = 200, 5000
X = rng.normal(size=(n, p))
y = rng.integers(0, 2, n)                      # no relationship whatsoever

selected = SelectKBest(f_classif, k=20).fit_transform(X, y)   # sees all rows
print("select first, then CV:",
      round(cross_val_score(LogisticRegression(max_iter=2000), selected, y, cv=5).mean(), 4))

pipe = Pipeline([("sel", SelectKBest(f_classif, k=20)),
                 ("clf", LogisticRegression(max_iter=2000))])
print("selection inside CV  :",
      round(cross_val_score(pipe, X, y, cv=5).mean(), 4))
# select first, then CV: 0.795
# selection inside CV  : 0.51

The features and labels were generated independently. Selecting the 20 best of 5000 features using every row, then cross-validating, reports 79.5% accuracy. Selection inside each fold gives 51.0%, close to the 50% accuracy expected on independent data with random labels.

With 5000 candidates and 200 rows, some features correlate with the labels by chance. Global selection uses each evaluation fold’s labels to decide which features survive, so those folds no longer provide an independent check. This does not require every selected feature to correlate with the labels in every fold. The amount of inflation depends on the candidate features, sample size, selector, and fitted model.

Leak two: rows that belong together

When several rows come from the same entity, such as a patient, user, or device, a random split can place that entity in both training and evaluation. A model may then rely on recognizing the entity. The following example gives each of 40 subjects 12 rows: 480 observations, but only 40 distinct subjects. Each subject has a persistent fingerprint, a fixed label, and additional noisy features related to the label. GroupKFold keeps all rows from each subject in one fold.

import numpy as np
from sklearn.model_selection import cross_val_score, KFold, GroupKFold
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(2)
n_sub, per = 40, 12
groups = np.repeat(np.arange(n_sub), per)
label_per_subject = rng.integers(0, 2, n_sub)
y = np.repeat(label_per_subject, per)

fingerprint = (np.repeat(rng.normal(size=(n_sub, 6)), per, axis=0)
               + 0.05 * rng.normal(size=(n_sub * per, 6)))       # identifies the subject
signal = (y * 2 - 1)[:, None] * 0.35 + rng.normal(size=(n_sub * per, 2))
X = np.hstack([fingerprint, signal])

m = KNeighborsClassifier(3)
print("plain KFold:",
      round(cross_val_score(m, X, y, cv=KFold(5, shuffle=True, random_state=0)).mean(), 4))
print("GroupKFold :",
      round(cross_val_score(m, X, y, cv=GroupKFold(5), groups=groups).mean(), 4))
# plain KFold: 0.9625
# GroupKFold : 0.5313

A shuffled split reports 96.3% and a grouped split reports 53.1%, a difference of about 43 percentage points. In the shuffled split, nearby training rows can come from the same subject and carry its label. Grouping removes that route for classifying held-out subjects, revealing how little of the shuffled score transfers to new subjects in this example.

The deployment question decides which split is right. For new subjects, hold out whole subjects. For future observations from known subjects, consider both subject overlap and time order: training on later observations can still misrepresent the intended task. A shuffled within-subject evaluation is appropriate only if the information available in its training rows matches the prediction setting.

Leak three: the future

For forecasting, a shuffled split can fit on observations later than those being evaluated. A forward-chaining split trains on an earlier segment and evaluates on a later one, then expands the training segment. The example below also compares the fitted model with predicting the last observed value. Ridge is a linear regression with a penalty on large coefficients. Its default score is \(R^2\), whereas the later comparison uses mean absolute error (MAE), the average absolute difference between prediction and outcome. A lower MAE is better.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score, KFold, TimeSeriesSplit
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(0)
level = np.cumsum(rng.normal(size=1200))       # random walk with independent increments
X = np.column_stack([np.roll(level, k) for k in (1, 2, 3, 4, 5)])[10:]
y = level[10:]

print("shuffled KFold R2 :",
      round(cross_val_score(Ridge(), X, y, cv=KFold(5, shuffle=True, random_state=0)).mean(), 4))
print("TimeSeriesSplit R2:",
      round(cross_val_score(Ridge(), X, y, cv=TimeSeriesSplit(5)).mean(), 4))

model_err, baseline_err = [], []
for tr, te in TimeSeriesSplit(5).split(X):
    fit = Ridge().fit(X[tr], y[tr])
    model_err.append(mean_absolute_error(y[te], fit.predict(X[te])))
    baseline_err.append(mean_absolute_error(y[te], X[te][:, 0]))   # predict last value
print(f"model MAE {np.mean(model_err):.4f}   last-value baseline MAE {np.mean(baseline_err):.4f}")
# shuffled KFold R2 : 0.9966
# TimeSeriesSplit R2: 0.9469
# model MAE 0.8001   last-value baseline MAE 0.7781

The forward-chaining split reports \(R^2 = 0.947\), but the model’s mean absolute error of 0.800 is worse than the last-value baseline’s 0.778. Each input row contains the five preceding observed levels: to predict \(y_t\), it uses \(y_{t-1},\ldots,y_{t-5}\). The initial slice removes the rows where np.roll would wrap values from the end of the series into the beginning. This evaluates one-step-ahead prediction: the model remains fixed within a test segment, while new actual observations become available as inputs. It does not forecast the entire segment from its starting point without observing intermediate values.

The next increment is independent noise, but the current level is useful for predicting the next level. Because the series wanders over a wide range, even the last-value forecast can explain much of its variation. The high \(R^2\) therefore does not establish an improvement over that baseline. Compare model and baseline using the same forecast horizon, available information, and metric; here that comparison uses MAE.

The two reported \(R^2\) values are means of fold scores, each computed as \(1-\sum_i(y_i-\hat y_i)^2/\sum_i(y_i-\bar y_{\mathrm{fold}})^2\). The denominator measures variation within that evaluation fold. Shuffled folds and contiguous time segments cover different ranges of the wandering series; they also use different training sizes. Their score difference therefore does not isolate a numerical amount of leakage. The time-ordered design is appropriate here because it respects the forecasting task.

A time split must respect when labels become available as well as when inputs arrive. For a target such as a customer’s spending over the next 30 days, recent training rows may still have unfinished outcomes at the first evaluation date. Exclude those rows until their labels would be known. A gap between training and evaluation can help, but its size must follow the target horizon and reporting delay. Overlapping lag inputs alone do not require a gap when all the shared observations are already available, as in this one-step example.

A taxonomy

These mechanisms need different checks. A pipeline controls where preprocessing is fitted, but it cannot decide whether subjects should be held out together or whether a field existed at prediction time. Leakage can occur without a feature being calculated directly from the target. Fix an invalid information path even if one comparison shows little change in score.

Check feature availability against the actual prediction time. A ticket’s resolution_category, an order’s refund_amount, or a patient’s discharge_diagnosis may be present in the warehouse but unavailable for an earlier prediction. Ordinary cross-validation on that table does not establish that the fields were available. Audit when values were written or revised, and reconstruct inputs as they would have appeared at prediction time. A later field need not reveal the answer perfectly to invalidate the evaluation.

Exercises

1. Target encoding on a meaningless identifier. Encode a 400-category column by its target mean using all rows, then repeat with the encoding computed inside each fold. The column has no relationship to the label.

Target encoding replaces a category with an average of its labels. For labels [1, 0, 1], that average is \(2/3\); if the last row is held out, the other two labels give \(1/2\). The difference shows how a row’s own answer can enter its input. Both versions below use the same five splits; categories absent from a training split receive that split’s overall label mean.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, KFold

rng = np.random.default_rng(0)
n = 2000
cat = rng.integers(0, 400, n)                  # 400 categories, ~5 rows each
y = rng.integers(0, 2, n)                      # independent of cat

splits = list(KFold(5, shuffle=True, random_state=0).split(cat))
means = np.full(400, y.mean())
for c in np.unique(cat):
    means = y[cat == c].mean()
print("encode on all data, then CV:",
      round(cross_val_score(LogisticRegression(), means[cat].reshape(-1, 1), y, cv=splits).mean(), 4))

scores = []
for tr, te in splits:
    m = np.full(400, y[tr].mean())             # fold-local encoding
    for c in range(400):
        rows = cat[tr] == c
        if rows.sum():
            m = y[tr][rows].mean()
    fit = LogisticRegression().fit(m[cat[tr]].reshape(-1, 1), y[tr])
    scores.append(fit.score(m[cat[te]].reshape(-1, 1), y[te]))
print("encode inside each fold    :", round(np.mean(scores), 4))
# encode on all data, then CV: 0.6855
# encode inside each fold    : 0.526

2. Compare global and fold-local scaling. Fit a StandardScaler on all the data and compare it with a scaler inside the pipeline, using the same splits on heavy-tailed features. Report the score gap and what one run can establish.

Check whether the score rises or falls, and convert the difference into a number of correct predictions out of 60.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(1)
n = 60
X = rng.standard_t(df=1.5, size=(n, 10))       # heavy tails: unstable mean and sd
y = (X[:, 0] > 0).astype(int)

leaky = cross_val_score(KNeighborsClassifier(3),
                        StandardScaler().fit_transform(X), y, cv=5).mean()
clean = cross_val_score(Pipeline([("s", StandardScaler()),
                                  ("c", KNeighborsClassifier(3))]), X, y, cv=5).mean()
print(f"scale first {leaky:.4f}   scale inside {clean:.4f}   gap {leaky - clean:+.4f}")
# scale first 0.6000   scale inside 0.6167   gap -0.0167

3. The search itself needs a held-out layer. Grid-search an SVM on pure noise and compare the best inner-CV score against a nested cross-validation estimate, averaged over 40 datasets.

Compare each average with the 50% accuracy expected on independent random labels. The nested score evaluates the search procedure on held-out rows.

In one outer split, 90 of the 120 rows are available for model development and 30 are held out. The inner four-fold search uses only those 90 rows to choose among 20 settings. Here C controls the SVM’s trade-off between regularization and training violations, and gamma controls how locally its radial-basis kernel compares inputs. Five values of C times four values of gamma give 20 candidates. It then refits the chosen setting on all 90 rows and scores the outer 30. Repeating the outer splits evaluates that search-and-fit procedure. A final model trained on all 120 rows uses more data than the models evaluated by the outer folds.

import numpy as np
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
from sklearn.svm import SVC

rng = np.random.default_rng(0)
grid = {"C": [0.01, 0.1, 1, 10, 100], "gamma": [0.001, 0.01, 0.1, 1]}
inner = KFold(4, shuffle=True, random_state=1)
outer = KFold(4, shuffle=True, random_state=2)

best, nested = [], []
for _ in range(40):
    X = rng.normal(size=(120, 40))
    y = rng.integers(0, 2, 120)                # no signal
    best.append(GridSearchCV(SVC(), grid, cv=inner).fit(X, y).best_score_)
    nested.append(cross_val_score(GridSearchCV(SVC(), grid, cv=inner),
                                  X, y, cv=outer).mean())

print(f"best inner-CV score  {np.mean(best):.4f}   optimism {np.mean(best) - 0.5:+.4f}")
print(f"nested CV score      {np.mean(nested):.4f}   gap from 0.5 {np.mean(nested) - 0.5:+.4f}")
# best inner-CV score  0.5519   optimism +0.0519
# nested CV score      0.5008   gap from 0.5 +0.0008