Expanding Window Validation
The first fold trained on 200 rows scores 1.77 and the last on 1,000 scores 0.87. Their mean summarizes this sequence of historical evaluations.
-
-
Data Leakage
Leakage can distort evaluation through future features, preprocessing, or test feedback. Time and group splits help only when they match the intended task.
-
Stratified Group K-Fold
A chance-level problem scored 0.9198 when groups were split across folds and 0.5323 when they were kept together. The truth was 0.5000.
-
Stratified Shuffle Split
Five splits of 1,000 rows tested 1,000 draws — but only 679 distinct rows. The other 321 were never in a test set at all.
-
Cross-Validation (CV)
On one dataset, five-fold CV ranges from 0.47 to 0.62 across split seeds. A separate ten-fold simulation illustrates why naive fold error bars can be too small.
-
Hyperparameter
Trying 300 configurations instead of one gained 0.0267 on the validation set and only 0.0082 in reality — and past 30, the real score fell.
-
Data Leakage and Train-Validation-Test Splits
Leakage compromises evaluation when unavailable information enters fitting or selection; it need not raise every observed score.