Holdout Reuse

Holdout reuse occurs when results from a reserved evaluation set influence later choices of models, features, metrics, or reported results. The selected system has then adapted to information from that set, even if no test row was directly used for fitting.

The scikit-learn cross-validation guide and Google’s dataset-splitting lesson describe the risk: test-driven tuning can fit accidental properties of the evaluation sample. A score used for selection is no longer an independent assessment of the selected model. The distortion depends on the search and data; it is not a fixed penalty for each look.

Four ways reuse accumulates

1. Tuning. Hyperparameters chosen by test performance.
2. Feature decisions. A feature kept because the test score improved when it was added.
3. Model selection. Reporting the best of many noisy estimates can favor positive estimation errors. This is an expected selection effect, not guaranteed inflation in every run; candidates with identical predictions add no new variation. See Hyperparameter Tuning and Model Selection.
4. Choosing what to report. The metric, slice, or run that looks best is the one presented.

These decisions can be made in good faith and still compromise evaluation. A single decision can matter substantially, while repeated looks need not produce a strictly increasing bias. Fresh data may yield a lower, similar, or higher score; the problem is losing an independent check of the selected system.

Separate the set you learn from and the set you trust

Use validation data or cross-validation within the development set to choose the model and its settings. Then fix the candidate and evaluation rules before opening the final test. Train, Validation, and Test Sets without Data Leakage develops the split. Independence from selection is one requirement; the final sample must also represent the intended task and support the stated uncertainty analysis.

• Seal it. Keep the final set where the modeling team cannot query it casually, and evaluate on it through a step that records each use.
• Fix the decision before the test. One look per release candidate is insufficient if the result influences the next candidate. For repeated confirmatory decisions, specify an appropriate multiple-testing or sequential procedure; an access budget alone does not supply one.
• Refresh when needed. If test results have informed development, confirm the revised system on representative data not used in those decisions. Previously reserved data can qualify; newly collected data are not automatically representative. Resplitting already exposed rows does not erase their influence.
• Report the history. State how many variants were compared on which set, so a reader can judge how much optimism the number may carry. How uncertain a single test result is to begin with is discussed in Comparing Models: Uncertainty and Significance.

These rules belong in the written plan for an evaluation, described in evaluation protocol. How reuse sits among the other ways an evaluation stops telling the truth is worked through in Ninety-Seven on the Test Set.

References:

• scikit-learn User Guide, Cross-validation: evaluating estimator performance — the risk of overfitting on the test set during tuning.
• Google for Developers, Machine Learning Crash Course: Dividing the original dataset (Creative Commons Attribution 4.0).