Blocked Splits (Single Holdout)

A blocked split is the simplest time-aware arrangement: order the data by time, train on the first portion, test on the rest. Eighty percent and twenty percent is the usual choice. There is one fold, one model and one number.

It is the right shape for the problem — nothing from the future enters training — and it is the least of the time-based splits, because that one number rests entirely on one arbitrary decision.

The cut point is the result

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(15)
n = 1200
t = np.arange(n)
season = np.sin(2 * np.pi * t / 200)
x = np.column_stack([season, np.roll(season, 5), t / n])
y = x @ np.array([4.0, 1.0, 2.0]) + rng.normal(scale=1.0, size=n) + 3.0 * (t > 900)
"""A level shift at row 900 — the kind of event a single holdout either straddles or misses."""

scores = []
for cut in range(700, 1101, 50):
    model = Ridge().fit(x[:cut], y[:cut])
    mae = mean_absolute_error(y[cut:], model.predict(x[cut:]))
    scores.append(mae)
    print(f"train on the first {cut:4d} rows, test on the rest  MAE {mae:.4f}")
print(f"one series, one model: MAE ranges {min(scores):.4f} to {max(scores):.4f}")
# train on the first  700 rows, test on the rest  MAE 2.1546
# train on the first  750 rows, test on the rest  MAE 2.3345
# train on the first  800 rows, test on the rest  MAE 2.5361
# train on the first  850 rows, test on the rest  MAE 2.8008
# train on the first  900 rows, test on the rest  MAE 3.1779
# train on the first  950 rows, test on the rest  MAE 2.4895
# train on the first 1000 rows, test on the rest  MAE 2.0125
# train on the first 1050 rows, test on the rest  MAE 1.5257
# train on the first 1100 rows, test on the rest  MAE 1.1867
# one series, one model: MAE ranges 1.1867 to 3.1779

One series, one model, nine defensible places to cut. The reported MAE runs from 1.1867 to 3.1779 — a factor of nearly three, with nothing varying but the boundary.

The shape is not random. This series has a level shift at row 900. Cutting at 900 is worst, because the test period then consists almost entirely of the new regime while training saw none of it. Cutting later puts part of the shift into training and the error falls away quickly. Cutting earlier gives a long test period in which the shift is only part of what is being scored, so it averages out.

So choosing the cut is choosing how hard a period to be evaluated on, and in practice that choice is made by a round number rather than by an argument. Someone reporting 1.19 and someone reporting 3.18 have both run a correct blocked split on the same data.

What it is still good for

• A first baseline. One fit answers whether the model is anywhere near useful, before spending on repeated folds.
• A final check on a held-back period. Keep its results out of model and preprocessing decisions until the evaluation is fixed. Reading an unchanged result again is not leakage; using it to revise the system and then presenting it as an independent final evaluation is the problem.
• When the deployment question really is about one period. “How would this have done last quarter” is a blocked split by definition.

A blocked holdout can compare models for the period it represents. The relevant quantity is their paired performance difference on that period, not one model’s absolute cut-point sensitivity. Check other periods when deployment spans different regimes, and report uncertainty and any ranking changes rather than dismissing a fixed numerical margin in advance.

Reporting one

Say where the cut is in calendar terms, not just as a percentage, and say what the test period contains — which seasons, which known events, whether anything unusual happened. A reader who knows the test window covered a promotion, an outage or a policy change can weigh the number; one who sees only “80/20 split” cannot.

And where the label looks forward, leave a gap between the two blocks at least as long as the label horizon, or the last training rows carry labels computed from the test period.

Repeated evaluation with cross-validation can use an expanding window or a sliding window to show how performance varies across forecast origins.

References

• scikit-learn User Guide, Cross-validation: evaluating estimator performance
• scikit-learn API reference, TimeSeriesSplit