Expanding Window Validation

An expanding window keeps every past observation and moves the test period forward. Fold one trains on 2020 and tests on 2021; fold two trains on 2020–2021 and tests on 2022; and so on. Once a row enters the training set it stays there.

This is the default arrangement in scikit-learn’s TimeSeriesSplit, which documents it directly: successive training sets are supersets of those that came before. Its premise is that old data still carries information — if the relationship you are modeling has not changed, throwing away history only costs precision.

The folds are not comparable

k-fold’s folds differ only in which rows they hold out; every model sees the same amount of data. Expanding-window folds differ in training size as well, and that shows up in the scores.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_absolute_error

def run(n_features, label):
    rng = np.random.default_rng(9)
    n = 1200
    x = rng.normal(size=(n, n_features))
    y = x @ rng.normal(size=n_features) + rng.normal(scale=1.0, size=n)
    scores = []
    for train, test in TimeSeriesSplit(n_splits=5).split(x):
        model = Ridge().fit(x[train], y[train])
        scores.append((len(train), mean_absolute_error(y[test], model.predict(x[test]))))
    print(label + "  " + "  ".join(f"{n_tr}:{mae:.3f}" for n_tr, mae in scores)
          + f"   mean {np.mean([m for _, m in scores]):.3f}")

run(3, "3 features (never starved) ")
run(150, "150 features (early folds starved)")
# 3 features (never starved)   200:0.836  400:0.869  600:0.791  800:0.872  1000:0.767   mean 0.827
# 150 features (early folds starved)  200:1.773  400:0.991  600:0.964  800:0.886  1000:0.870   mean 1.097

With 150 features, the first fold trains on 200 rows and posts MAE 1.773; the last trains on 1,000 and posts 0.870. The mean 1.097 summarizes those historical fits and periods. The final refit uses a different training size and faces a new period, so neither the mean nor the last fold guarantees its future performance.

The first line shows the effect is conditional. With three features, 200 rows is already plenty, and the fold scores bounce between 0.767 and 0.872 with no relation to training size at all. So “early folds drag the average down” is not a property of the method — it is what happens when the early folds are data-starved, which depends on the model’s appetite rather than on the calendar.

Either way, report the per-fold numbers with their training sizes and test periods. A single mean over an expanding window hides both the learning curve and which periods were hard.

When the premise holds

The choice reflects assumptions about history and the deployment task, and it can be tuned using a validation design that respects that task. Keeping all history says old data is still representative; a sliding window says it is not, and the cost of being wrong runs in both directions — measured there.

Two practical notes. Set gap when the label looks forward, so the last training rows do not have labels computed from the test period. And where the series is long enough to support it, running both arrangements and comparing is cheap and settles the question empirically rather than by argument — see time-based splits.

References

• scikit-learn API reference, TimeSeriesSplit
• scikit-learn User Guide, Cross-validation: evaluating estimator performance