Sliding Window (Rolling Window) Cross-Validation

A sliding window trains on a fixed span that moves forward with the test period. With a three-month window: train on January–March and test April, then train on February–April and test May, then March–May and test June. Each step adds a new month and drops the oldest.

That dropping is the whole difference from an expanding window, which keeps everything. In scikit-learn it is TimeSeriesSplit(max_train_size=...).

Forgetting as the feature

Discarding data sounds like a cost, and it is one whenever the relationship being modeled is stable. It becomes the point as soon as the relationship changes.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(12)
n, change_at = 1200, 700
t = np.arange(n)
coef = np.where(t[:, None] < change_at, np.array([3.0, 1.5]), np.array([-2.0, 4.0]))
"""The relationship flips at row 700 — a regime change, not noise."""
x = rng.normal(size=(n, 2))
y = (x * coef).sum(axis=1) + rng.normal(scale=0.5, size=n)

window = 300
for start in (800, 950, 1100):
    test = slice(start, start + 100)
    expanding = Ridge().fit(x[:start], y[:start])
    sliding = Ridge().fit(x[start - window:start], y[start - window:start])
    print(f"test rows {start}-{start+100}  expanding MAE {mean_absolute_error(y[test], expanding.predict(x[test])):.4f}"
          f"   sliding({window}) MAE {mean_absolute_error(y[test], sliding.predict(x[test])):.4f}")
# test rows 800-900  expanding MAE 4.0757   sliding(300) MAE 3.1584
# test rows 950-1050  expanding MAE 2.9190   sliding(300) MAE 0.7958
# test rows 1100-1200  expanding MAE 2.7898   sliding(300) MAE 0.4347

The coefficients flip at row 700 — a competitor enters, a policy changes, a sensor is replaced. Immediately after, both approaches are poor and the sliding window is only somewhat better, because its 300-row window still contains mostly pre-change data. Four hundred rows later the window has cleared and the sliding model reaches an MAE of 0.4347, while the expanding model stalls at 2.7898.

Within the observed post-change period, the expanding fit still gives substantial weight to the 700 pre-change rows and adapts slowly. This is not permanent inability to recover: if the new regime persists, the old rows can become a negligible fraction. The example measures the cost of retaining them over this particular horizon.

Choosing the window

The length is a bias–variance trade made explicit:

• Short. Adapts within a few periods of a change, and every estimate rests on little data, so the fold-to-fold scores are noisy and the model is prone to chasing a blip.
• Long. Steadier estimates and slower adaptation; in the limit it becomes an expanding window and inherits its behavior above.
• At least one seasonal cycle, if the model has to learn seasonality at all. A window shorter than the period cannot see it, and the model will attribute the seasonal swing to whatever else moved.

Tuning the length against the validation folds makes it a hyperparameter like any other, with the same consequence: the score that chose it is no longer an unbiased estimate of the chosen configuration. Where that matters, hold back a final period the search never touched.

One thing it does better than the alternative

A fixed window holds training size roughly constant, making one source of score differences easier to interpret. Test periods can still differ in season, difficulty, noise and distribution. Rising errors may prompt a drift investigation, but do not identify drift by themselves. Both sliding and expanding scores need their dates and operating context alongside them.

Holding training size constant can aid diagnosis, but does not make sliding windows universally better diagnostics. The usual rules still apply: leave a gap when labels look forward, and fit every preprocessing step inside the window — see time-based splits.

References

• scikit-learn API reference, TimeSeriesSplit
• scikit-learn User Guide, Cross-validation: evaluating estimator performance