Time-Based Splits

A time-based split keeps the training data strictly before the test data. Every fold obeys one rule — the training period ends before the validation period begins — so the model is never fitted on something that had not happened yet.

Shuffled k-fold can put August in the training set while forecasting July. Even without shuffling, ordinary k-fold trains on the other folds, which can include later observations; it does not enforce training before testing. On temporal data, that evaluates a different prediction setting.

Check the target before the split

Time-based splitting is usually presented as the fix for time-series evaluation. It is necessary and it is not where most inflated numbers come from.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

rng = np.random.default_rng(5)
series = np.cumsum(rng.normal(size=600))
"""A random walk. The level is highly autocorrelated; the step is pure noise."""
lags = np.column_stack([np.roll(series, k) for k in range(1, 6)])[5:]
level = series[5:]
change = np.diff(series)[4:]

model = RandomForestRegressor(n_estimators=60, random_state=0)
shuffled = KFold(5, shuffle=True, random_state=0)
chronological = TimeSeriesSplit(5)
for name, target, rows in (("predict the level ", level, slice(None)),
                           ("predict the change", change, slice(0, -1))):
    a = cross_val_score(model, lags[rows], target[:lags[rows].shape[0]], cv=shuffled).mean()
    b = cross_val_score(model, lags[rows], target[:lags[rows].shape[0]], cv=chronological).mean()
    print(f"{name}  shuffled KFold R2 {a:+.4f}   TimeSeriesSplit R2 {b:+.4f}")
# predict the level   shuffled KFold R2 +0.9667   TimeSeriesSplit R2 +0.8309
# predict the change  shuffled KFold R2 -0.1195   TimeSeriesSplit R2 -0.2246

The series is a random walk, so its increments are unpredictable by construction. Asked to predict the level from the last five levels, a random forest scores \(R^2 = 0.83\) under a proper chronological split — and 0.97 under a shuffled one. Asked to predict the change, the same model scores below zero under both.

The level R² of 0.83 does not establish that the model beats simply predicting the last level. A level forecast can be exactly what a demand or inventory decision needs; evaluating changes is not compulsory for every forecasting task. Fix the target and horizon from the use case, compare with an appropriate naïve baseline on that same target, and use change prediction as an additional diagnostic when it addresses the question.

The three arrangements

The folds these produce are not exchangeable the way k-fold’s are. A later fold trains on more data and covers a different period, so averaging the five scores mixes quantities that differ for reasons other than the model. Report the per-fold numbers with their training sizes and dates alongside any mean.

Chronological is not always enough

A split can respect time and still leak, whenever a row’s features or label reach across the boundary:

• Forward-looking labels. For a 30-day target, purge training rows whose label interval or label availability crosses the prediction boundary. Thirty rows equal thirty days only in daily, equally spaced data. TimeSeriesSplit.gap counts samples, so choose it from timestamps and label availability rather than copying a calendar horizon as a row count.
• Features built with unavailable observations. A centered window or backward fill can read future observations. Forward fill normally carries a past observation forward and is not inherently future leakage, although the source value must actually have arrived by prediction time.
• Anything fitted on the whole series. A scaler, a seasonal decomposition or a target encoding built before the split has seen every period — the same failure demonstrated under k-fold cross-validation.
• Late-arriving data. A feature that was only known three days after the timestamp it carries is not available at prediction time, however early its timestamp is.

The last one is not a splitting problem at all and no splitter can detect it. It is answered by asking, for every feature, what was actually known at the moment the prediction would have been made.

References

• scikit-learn API reference, TimeSeriesSplit
• scikit-learn User Guide, Cross-validation: evaluating estimator performance