Model Evaluation Reliability

In this fictional example, a claims triage model had been in development for four months when it went to the review board with an area under the ROC curve of 0.97. It was meant to flag filed insurance claims likely to need an adjuster’s investigation. In its first month in production the same measure, computed only on claims already resolved, was 0.71. That was a provisional score for the resolved subset: slower cases were still missing, so it did not yet describe the entire month’s claims.

The investigation found four problems. One feature recorded which team had last updated the claim’s status, and in historical data that field was usually filled in after an adjuster had already looked, so it partly encoded the answer. The data had been split at random, while the model would score next month’s claims, so the test set contained claims from the same weeks — and the same storms — as the training set. A large broker’s claims had been submitted in near-duplicate batches that landed on both sides of the split. And over four months the team had tried about sixty variants of features and settings, each time keeping whichever did best on the same test set.

These problems invalidated the score as an independent assessment of the intended task. A single leak can cause a large distortion; neither its size nor an increase in every observed score is guaranteed. An evaluation score estimates performance for a defined population and prediction setting. Its reliability depends on how well the design represents that setting, how outcomes are observed, and how much uncertainty remains.

An evaluation is a rehearsal of deployment

The scikit-learn cross-validation guide separates data used for fitting and selection from data reserved for final evaluation. That protects against one failure: evaluation information influencing the model. Other failures need different checks. A fully independent test can still represent the wrong population, contain too few informative cases, or use a metric that does not answer the decision question.

Three axes decide whether the rehearsal is faithful. Time: the model will predict the future from the past, so it should be tested that way. Population: if it will meet new customers or sites, the test should contain ones it has not met. Information: every input must be something known at the moment the prediction is made. Writing down the deployment conditions on these three axes before modeling starts is the core of an evaluation protocol, and the conditions themselves come from the problem framing.

Overfitting to the training data

Overfitting is excessive adaptation to the training sample that harms performance on the intended prediction task. Google’s introductory treatment illustrates it through memorization, but memorizing every row is not required. A growing training–validation gap is a reason to investigate, not a diagnosis by itself: distribution differences, small samples, and different training and evaluation settings can also produce a gap.

Compare learning curves under comparable measurement conditions, including how augmentation and dropout are used. A representative independent holdout helps reveal generalization problems, but a finite test can miss them. The theory is developed in Empirical Risk Minimization and Generalization and The Bias-Variance Decomposition, Derived and Measured, and the practical diagnosis for neural networks in Diagnosing and Preventing Overfitting in Deep Learning.

Leakage: information the model will not have

Form	What happens	In the opening
Future information	A feature’s value was recorded after the moment of prediction, often because of the outcome itself	The team that last updated the claim status
Duplicates	The same example, or a near copy, appears in both training and test data	The broker’s batch submissions
Related rows	Shared entities or events can make evaluation easier than deployment when the task requires generalization to unseen groups	Claims from the same storm on both sides when the target is future storms

For every feature, ask when its value became available to the prediction system. Reconstruct that available version, rather than a later correction to historical data, using a point-in-time join. Check labels separately: define the outcome window, when an outcome is considered final, and how unresolved cases are handled. At each simulated training cutoff, use only labels that would then have been available. These information boundaries and duplicate handling are developed in Train, Validation, and Test Sets without Data Leakage.

The split has to match how the model will be used

Random splitting is a procedure; it does not make dependent observations independent. The scikit-learn guide explains why common IID evaluation assumptions can fail for grouped or time-dependent data. A shuffled split may use later observations or shared group information unavailable in the intended task. For new entities, hold out whole groups. For future observations from known entities, overlap can be appropriate, provided the time and information boundaries match deployment.

The rule that follows is to choose the split by asking how the model will be used, not by what the tooling does by default. A model that scores next month’s claims is tested on a later month. A model that will serve new stores is tested on stores it never saw. A forecasting model is backtested on the same schedule it will run, and even then the features need separate checking, points developed in Time-Series Forecasting.

Overfitting to the test set

In the opening, the team used the same test results to choose sixty successive variants. Each comparison fed evaluation information into development. As the scikit-learn guide and Google’s dataset-splitting lesson explain, a set used this way no longer provides an independent final assessment of the selected model.

Choose features, settings, and the candidate on development data, then evaluate the fixed candidate on an untouched final set. One look per candidate does not protect a set if the result informs the next candidate. Repeated confirmatory comparisons require an appropriate prespecified multiple-testing or sequential design; a count of permitted looks alone is insufficient. Selecting among noisy estimates can favor positive errors, but it does not force inflation in every run, especially when candidates make identical predictions. Hyperparameter Tuning and Model Selection develops that distinction; holdout reuse sets out the operational rules.

Bias in what the data represents

A perfectly split evaluation still measures the population actually observed. Google’s treatment of reporting bias describes how recorded events can differ from the population of interest. For a model that triages filed claims, people who never filed are outside the stated target; their absence becomes a limitation if the result is extended to all potential claims. Within filed claims, restricting evaluation to already resolved cases can create a different selection problem if resolution time is related to the outcome or prediction difficulty.

The useful habit is to trace how each record came to exist: which process created it, who was never recorded, and whose outcome was only observed because of a past decision. The general patterns are described in selection and nonresponse bias and survivorship bias. Where the concern is unequal performance across groups of people, the criteria and audits are covered in Fairness and Bias Auditing.

Imbalance

When a class is rare, high accuracy can coexist with missing nearly every positive case. Accuracy still measures overall correctness, but it needs a majority-class baseline and class-specific measures. If the rare class is designated positive, precision and recall describe different aspects of its detection; neither alone is a general error-cost function. Misclassification Cost distinguishes costs and decision rules. Class Imbalance explains the evaluation consequences, and Imbalanced Data develops weighting and sampling methods. Training rebalancing can change probability interpretation, so check calibration on representative independent data before deciding whether correction is needed; see Probability Calibration. Keep training resampling within training splits. A deliberately stratified evaluation sample can be valid when its sampling design, target-distribution adjustments, and uncertainty are handled explicitly; its raw balanced-sample score is not automatically an operating score.

A single number hides its uncertainty

Plan test size around the precision required or the power to detect a specified effect at a chosen significance level. No sample size guarantees a significant result, and a precise finding of little difference can be useful. Count the observations relevant to the metric: actual positives for recall, predicted positives for precision, both classes for AUC. Grouping and time dependence also affect how much information those counts contain.

Report a point estimate with an appropriate interval and explain what varies. Variation across folds, variation across retraining seeds, and test-sampling uncertainty measure different things. Compare fixed candidates on the same cases using paired inference for their difference, accounting for dependence between cases when needed. Comparing Models: Uncertainty and Significance explains the methods. An improvement smaller than one model’s standard error can still be detectable in a paired comparison; a nonsignificant result does not establish zero improvement. Assess the difference and its uncertainty against a prespecified threshold of practical value.

Write the protocol before the model

A protocol makes the intended prediction setting and assessment rules reviewable before results influence development. It fixes the prediction moment and population, label availability and observation windows, split scheme, deciding metric, baselines, test-size rationale, final-set access, reported slices, and uncertainty methods. Its fields are listed in evaluation protocol.

A reviewer can question a result with or without a written protocol. The protocol adds a record of what was decided before results were visible, making departures from the original plan easier to identify and assess.

What a reviewer should ask

• Does the split imitate deployment in time, population, and information?
• When were features and training labels available, and how are unresolved evaluation outcomes handled?
• Does duplicate and group handling match the task: new entities, known entities over time, or both?
• How many times was the final test set used, and for what decisions?
• What process produced the data, and who is missing from it?
• How many rare cases are in the test set, and what is the interval on the rare-class metrics?
• What is the paired difference from the baseline, its uncertainty, and the minimum improvement worth acting on?
• Were the rules fixed before results were seen? If results prompted changes, what independent confirmation or suitable adaptive inference supports the revised claim?

A high score alone does not establish a sound evaluation. Neither does shuffled cross-validation for a task requiring future prediction, or reporting only the winner of an unaccounted search. Each needs evidence that the data, selection procedure, and inference match the claim.

In the fictional example, the team removed the status field, split by month, and kept duplicate broker batches together. After fixing the candidate, it evaluated once on an untouched recent-quarter cohort and obtained 0.79.

The next evaluation review should report the outcome window, how unresolved cases were handled, the sample size, and uncertainty alongside 0.79. Compare the repaired candidate with the baseline under the same conditions. Because 0.97, 0.71, and 0.79 were measured under different conditions, their differences cannot isolate the effect of an individual repair. Whether the model is worth deploying depends on the baseline and error costs, as discussed in Predicting the Wrong Thing Well.

References:

• scikit-learn User Guide, Cross-validation: evaluating estimator performance — the methodological principle, overfitting on the test set, and when independence cannot be assumed.
• Google for Developers, Machine Learning Crash Course: Overfitting (Creative Commons Attribution 4.0).
• Google for Developers, Machine Learning Crash Course: Dividing the original dataset (Creative Commons Attribution 4.0).
• Google for Developers, Machine Learning Crash Course: Imbalanced datasets (Creative Commons Attribution 4.0).
• Google for Developers, Machine Learning Crash Course: Types of bias (Creative Commons Attribution 4.0).