Evaluation Protocol

An evaluation protocol is a written plan for judging a model, agreed before development results guide selection. It defines the prediction task, data splits, metrics, baselines, final-test access, and uncertainty reporting. Choosing these rules after seeing results can favor what happened to look good, so the timing of decisions matters.

The scikit-learn cross-validation guide explains why fitting and model selection must be separated from final evaluation. A protocol also addresses problems that do not require leakage: a population mismatch, incomplete outcomes, imprecise estimates, or a metric unrelated to the decision.

Imitate deployment on three axes

An evaluation is a rehearsal of deployment, and it is only as good as the rehearsal is faithful. Three axes matter.

• Time. Test future predictions using training information available at the simulated cutoff. This includes label availability as well as feature timestamps. Define the outcome window and treatment of outcomes still unresolved when evaluation is reported. Match retraining and forecast timing to deployment, as developed in Time-Series Forecasting.
• Population. For new customers, stores, or patients, hold out whole entities. For future predictions about known entities, overlap may be appropriate while respecting time order. Specify who enters the evaluation and whether outcome availability excludes part of that population.
• Information. Reconstruct the feature values actually available at prediction time. Prevent duplicate observations from supplying answers to an evaluation intended to test new cases, and keep test results out of fitting and selection. See Train, Validation, and Test Sets without Data Leakage. Google’s dataset-splitting lesson motivates these checks through representativeness and duplicate examples.

Eight things to write down

1. The prediction moment, target population, outcome window, and label availability, as fixed in the problem framing.
2. The split scheme and why it matches deployment on the three axes.
3. The deciding metric and practical success threshold, plus permitted deterioration in secondary metrics and how uncertainty affects each decision.
4. The baselines the model must beat: current practice and a simple predictor.
5. Test size justified by desired interval precision or by a specified effect, significance level, and power. Include metric-specific counts and the effects of grouped or time-dependent cases.
6. Who can access the final set, which decisions are fixed before it is opened, and what design supports any repeated confirmatory use; see holdout reuse.
7. Which uncertainty is being estimated: sampling of test cases, variation across refits, or a paired difference between candidates. Fold spread is not automatically a confidence interval; see Comparing Models: Uncertainty and Significance.
8. Slices to report separately: rare classes, important customer groups, recent periods. Rare-class issues are covered in class imbalance.

Assign an owner and keep a change log. A better split may justify revising the protocol, but results from different protocols are not one comparable series. If observed results prompted the revision or a new candidate, label that work exploratory. Confirmation requires newly fixed rules and representative data that have not influenced those decisions, or an inference procedure designed for the adaptation. Recording the change alone does not restore independence.

How a protocol protects against the ways evaluations mislead is worked through in Ninety-Seven on the Test Set.

References:

• scikit-learn User Guide, Cross-validation: evaluating estimator performance — the methodological principle, and when independence cannot be assumed.
• Google for Developers, Machine Learning Crash Course: Dividing the original dataset (Creative Commons Attribution 4.0) — criteria for a good test set and the duplicates problem.