Data Leakage and Train-Validation-Test Splits

Leakage occurs when model development uses information that would be unavailable for the intended prediction, or when data reserved for independent evaluation influence fitting or selection. It can make a model look useful for a task it has not actually been tested on. The examples below examine feature selection, repeated observations from the same subject, and forecasting with time-ordered data.

The rule

Training data are used to fit the model and its preprocessing. Validation data help choose settings or compare approaches. Test data provide a final evaluation after those choices are settled. A validation score used to choose the winner should not also be presented as an independent assessment of that winner. Reserve the test set before development, and choose a split that matches who or what the model will predict for.

For a model chosen independently of an i.i.d. evaluation sample from the target distribution, the average evaluation loss is an unbiased estimate of its risk, provided the expected loss is finite. Using evaluation data during fitting or selection removes that guarantee. The distortion depends on how the information affects the fitted model and its evaluation; it need not increase the score in every run. The empirical risk minimization and generalization article develops this distinction.

Within each training split, fit the feature selector, scaler, or encoder using only that split, then apply the fitted transformation to its held-out portion. Validation can choose the preprocessing method or its settings. A pipeline bundles these fitted steps with the model so cross-validation refits the whole procedure on each training portion. A transformation fitted on the entire dataset before cross-validation has already used its held-out rows. Fixed operations, such as converting centimeters to meters, do not estimate anything from the data and can be applied before splitting. For fitted transformations, including imputation and scaling, the training boundary matters.

Leak one: selecting features on all the data

A label-based feature screen can pick chance associations if it sees evaluation labels. Here SelectKBest uses f_classif to rank features by differences between the two classes and keeps 20. In five-fold cross-validation, each fold is a held-out group of 40 of the 200 rows. The pipeline selects features using the other 160 rows, fits the classifier there, and applies the same selected columns to the 40 held-out rows. It repeats this with each fold held out once. For this binary classifier, cv=5 uses stratified folds, keeping class proportions approximately similar, and cross_val_score reports accuracy by default. The printed value is the mean of the five fold accuracies. Stratification by itself does not keep subjects together or preserve time order.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
n, p = 200, 5000
X = rng.normal(size=(n, p))
y = rng.integers(0, 2, n)                      # no relationship whatsoever

selected = SelectKBest(f_classif, k=20).fit_transform(X, y)   # sees all rows
print("select first, then CV:",
      round(cross_val_score(LogisticRegression(max_iter=2000), selected, y, cv=5).mean(), 4))

pipe = Pipeline([("sel", SelectKBest(f_classif, k=20)),
                 ("clf", LogisticRegression(max_iter=2000))])
print("selection inside CV  :",
      round(cross_val_score(pipe, X, y, cv=5).mean(), 4))
# select first, then CV: 0.795
# selection inside CV  : 0.51

The features and labels were generated independently. Selecting the 20 best of 5000 features using every row, then cross-validating, reports 79.5% accuracy. Selection inside each fold gives 51.0%, close to the 50% accuracy expected on independent data with random labels.

With 5000 candidates and 200 rows, some features correlate with the labels by chance. Global selection uses each evaluation fold’s labels to decide which features survive, so those folds no longer provide an independent check. This does not require every selected feature to correlate with the labels in every fold. The amount of inflation depends on the candidate features, sample size, selector, and fitted model.

Leak two: rows that belong together

When several rows come from the same entity, such as a patient, user, or device, a random split can place that entity in both training and evaluation. A model may then rely on recognizing the entity. The following example gives each of 40 subjects 12 rows: 480 observations, but only 40 distinct subjects. Each subject has a persistent fingerprint, a fixed label, and additional noisy features related to the label. GroupKFold keeps all rows from each subject in one fold.

import numpy as np
from sklearn.model_selection import cross_val_score, KFold, GroupKFold
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(2)
n_sub, per = 40, 12
groups = np.repeat(np.arange(n_sub), per)
label_per_subject = rng.integers(0, 2, n_sub)
y = np.repeat(label_per_subject, per)

fingerprint = (np.repeat(rng.normal(size=(n_sub, 6)), per, axis=0)
               + 0.05 * rng.normal(size=(n_sub * per, 6)))       # identifies the subject
signal = (y * 2 - 1)[:, None] * 0.35 + rng.normal(size=(n_sub * per, 2))
X = np.hstack([fingerprint, signal])

m = KNeighborsClassifier(3)
print("plain KFold:",
      round(cross_val_score(m, X, y, cv=KFold(5, shuffle=True, random_state=0)).mean(), 4))
print("GroupKFold :",
      round(cross_val_score(m, X, y, cv=GroupKFold(5), groups=groups).mean(), 4))
# plain KFold: 0.9625
# GroupKFold : 0.5313

A shuffled split reports 96.3% and a grouped split reports 53.1%, a difference of about 43 percentage points. In the shuffled split, nearby training rows can come from the same subject and carry its label. Grouping removes that route for classifying held-out subjects, revealing how little of the shuffled score transfers to new subjects in this example.

The deployment question decides which split is right. For new subjects, hold out whole subjects. For future observations from known subjects, consider both subject overlap and time order: training on later observations can still misrepresent the intended task. A shuffled within-subject evaluation is appropriate only if the information available in its training rows matches the prediction setting.

Leak three: the future

For forecasting, a shuffled split can fit on observations later than those being evaluated. A forward-chaining split trains on an earlier segment and evaluates on a later one, then expands the training segment. The example below also compares the fitted model with predicting the last observed value. Ridge is a linear regression with a penalty on large coefficients. Its default score is \(R^2\), whereas the later comparison uses mean absolute error (MAE), the average absolute difference between prediction and outcome. A lower MAE is better.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score, KFold, TimeSeriesSplit
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(0)
level = np.cumsum(rng.normal(size=1200))       # random walk with independent increments
X = np.column_stack([np.roll(level, k) for k in (1, 2, 3, 4, 5)])[10:]
y = level[10:]

print("shuffled KFold R2 :",
      round(cross_val_score(Ridge(), X, y, cv=KFold(5, shuffle=True, random_state=0)).mean(), 4))
print("TimeSeriesSplit R2:",
      round(cross_val_score(Ridge(), X, y, cv=TimeSeriesSplit(5)).mean(), 4))

model_err, baseline_err = [], []
for tr, te in TimeSeriesSplit(5).split(X):
    fit = Ridge().fit(X[tr], y[tr])
    model_err.append(mean_absolute_error(y[te], fit.predict(X[te])))
    baseline_err.append(mean_absolute_error(y[te], X[te][:, 0]))   # predict last value
print(f"model MAE {np.mean(model_err):.4f}   last-value baseline MAE {np.mean(baseline_err):.4f}")
# shuffled KFold R2 : 0.9966
# TimeSeriesSplit R2: 0.9469
# model MAE 0.8001   last-value baseline MAE 0.7781

The forward-chaining split reports \(R^2 = 0.947\), but the model’s mean absolute error of 0.800 is worse than the last-value baseline’s 0.778. Each input row contains the five preceding observed levels: to predict \(y_t\), it uses \(y_{t-1},\ldots,y_{t-5}\). The initial slice removes the rows where np.roll would wrap values from the end of the series into the beginning. This evaluates one-step-ahead prediction: the model remains fixed within a test segment, while new actual observations become available as inputs. It does not forecast the entire segment from its starting point without observing intermediate values.

The next increment is independent noise, but the current level is useful for predicting the next level. Because the series wanders over a wide range, even the last-value forecast can explain much of its variation. The high \(R^2\) therefore does not establish an improvement over that baseline. Compare model and baseline using the same forecast horizon, available information, and metric; here that comparison uses MAE.

The two reported \(R^2\) values are means of fold scores, each computed as \(1-\sum_i(y_i-\hat y_i)^2/\sum_i(y_i-\bar y_{\mathrm{fold}})^2\). The denominator measures variation within that evaluation fold. Shuffled folds and contiguous time segments cover different ranges of the wandering series; they also use different training sizes. Their score difference therefore does not isolate a numerical amount of leakage. The time-ordered design is appropriate here because it respects the forecasting task.

A time split must respect when labels become available as well as when inputs arrive. For a target such as a customer’s spending over the next 30 days, recent training rows may still have unfinished outcomes at the first evaluation date. Exclude those rows until their labels would be known. A gap between training and evaluation can help, but its size must follow the target horizon and reporting delay. Overlapping lag inputs alone do not require a gap when all the shared observations are already available, as in this one-step example.

A taxonomy

Leak	How it happens	When it matters
Feature selection on all data	evaluation labels affect the selected columns	Many candidate features can supply chance associations.
Target encoding on all data	evaluation labels enter category averages	Small category counts give each included label more weight.
Grouped rows split randomly	the same entity appears on both sides	The intended task involves previously unseen entities.
Temporal order ignored	later observations influence earlier predictions	Deployment must predict using only information already available.
Duplicate or near-duplicate rows	copies of a record cross the split	The evaluation is meant to represent independent new cases.
A feature unavailable at prediction time	a field is populated or updated later	The historical value includes information the deployed model cannot use.
Scaling fitted on all data	evaluation rows affect the estimated scales	Its effect depends on the data and model; a small observed score gap does not establish that it is harmless.

These mechanisms need different checks. A pipeline controls where preprocessing is fitted, but it cannot decide whether subjects should be held out together or whether a field existed at prediction time. Leakage can occur without a feature being calculated directly from the target. Fix an invalid information path even if one comparison shows little change in score.

Check feature availability against the actual prediction time. A ticket’s resolution_category, an order’s refund_amount, or a patient’s discharge_diagnosis may be present in the warehouse but unavailable for an earlier prediction. Ordinary cross-validation on that table does not establish that the fields were available. Audit when values were written or revised, and reconstruct inputs as they would have appeared at prediction time. A later field need not reveal the answer perfectly to invalidate the evaluation.

References

• scikit-learn documentation. sklearn.model_selection.TimeSeriesSplit