Stratified Shuffle Split

Stratified shuffle split draws repeated random train/test pairs, keeping each draw’s class proportions close to the full dataset’s. It takes two parameters that k-fold ties together: n_splits, how many times to draw, and test_size, how large each test set is.

On 1,000 samples split 80/20 with five splits at test_size=0.2, each draw gives 800 training rows (640 and 160 by class) and 200 test rows (160 and 40). The class balance is preserved every time.

The splits overlap

The word “split” invites a wrong picture. k-fold partitions the data: every row sits in exactly one test fold and is tested exactly once. Shuffle split draws each test set independently, so a row can be drawn several times or never.

import numpy as np
from sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit

rng = np.random.default_rng(8)
n = 1000
y = (rng.uniform(size=n) < 0.2).astype(int)
x = np.zeros((n, 1))

kf = list(StratifiedKFold(5, shuffle=True, random_state=0).split(x, y))
ss = list(StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=0).split(x, y))
for name, splits in (("StratifiedKFold", kf), ("StratifiedShuffleSplit", ss)):
    seen = np.concatenate([test for _, test in splits])
    sizes = [len(test) for _, test in splits]
    rates = [round(float(y[test].mean()), 3) for _, test in splits]
    print(f"{name:23} test sizes {sizes}  positive rate {rates}")
    print(f"{'':23} distinct rows tested {len(set(seen.tolist())):4d} of {n}  total draws {len(seen)}")

never = n - len(set(np.concatenate([test for _, test in ss]).tolist()))
print(f"rows StratifiedShuffleSplit never tested: {never}")
# StratifiedKFold         test sizes [200, 200, 200, 200, 200]  positive rate [0.195, 0.195, 0.195, 0.195, 0.195]
#                         distinct rows tested 1000 of 1000  total draws 1000
# StratifiedShuffleSplit  test sizes [200, 200, 200, 200, 200]  positive rate [0.195, 0.195, 0.195, 0.195, 0.195]
#                         distinct rows tested  679 of 1000  total draws 1000
# rows StratifiedShuffleSplit never tested: 321

Both schemes produced five test sets of 200 with the same 19.5 percent positive rate, and both made 1,000 test draws in total. k-fold’s draws cover 1,000 distinct rows, because that is what a partition does. Shuffle split’s cover 679 — so 321 rows, nearly a third of the dataset, were never evaluated at all, while others were evaluated two or three times.

That is not a defect; it is what independent draws mean. But it changes what the average of the five scores is estimating. k-fold’s average is one estimate that uses every row once. Shuffle split’s average is the mean of repeated estimates on samples of a fixed size, and its splits share rows, so their scores are correlated and the average does not tighten with n_splits the way independent replicates would.

What the extra parameter buys

The practical case for it is exactly that decoupling. Wanting twenty rounds with k-fold means twenty folds and a 5 percent test set; shuffle split allows twenty rounds at 20 percent. On a large dataset where each test draw is already large enough to be precise, repeated draws give a cleaner read on how much the estimate moves with the split than a single k-fold pass does.

With scarce data, incomplete test coverage can be a disadvantage relative to a partition. The untested rows are nevertheless selected by the random splitting mechanism: within a class, rows have the same chance of never appearing in a test draw. Report coverage and distinguish uncertainty from repeated splits of one dataset from uncertainty over new datasets.

Where it helps most

The stratification does the same job here as everywhere: on imbalanced data it stops a test draw from landing with no minority examples, which is common enough to matter — see stratified cross-validation. With a rare class it is worth checking that test_size leaves enough positives in each draw to estimate recall from at all.

And as with every resampling scheme, any fitted preprocessing belongs inside the loop; drawing the splits differently does nothing about a scaler fitted on all the data first.

References

• scikit-learn API reference, StratifiedShuffleSplit
• scikit-learn User Guide, Cross-validation: evaluating estimator performance