Evaluation Set

An evaluation set contains examples and a basis for assessing model or system outputs. Its role must be stated. A development or validation set informs candidate choices; a regression suite checks known cases; an independent confirmation set supports assessment after those choices are fixed. Data used repeatedly for development can still be useful, but its selected score is not an independent estimate of performance on unseen cases.

Three roles

The usual arrangement divides labeled data into three parts. The training set fits the model’s parameters. The validation set, also called the development set, is used to compare settings and candidate models and to decide when to stop training. The test set is kept aside and used at the end to estimate how the chosen model will perform. With 100,000 labeled examples, a split of 70,000, 15,000, and 15,000 is a common starting point. When data is scarce, cross-validation on the training portion can take the validation role, so that more examples are available for fitting.

Training, validation, and test scores can prompt investigation. In an illustrative 98%, 91%, and 90% accuracy comparison, the training score is higher, but the cause depends on sample composition, uncertainty, selection history, and the fitted model. A small validation/test gap does not establish that validation was not overused or that leakage is absent. Inspect the design and access history before deciding whether regularization, new data, or a different evaluation is needed.

What makes the estimate trustworthy

An independent test estimate requires that candidate and reporting choices were fixed without its feedback, together with representative sampling and a valid uncertainty analysis. Selecting winners on test results can favor optimistic errors, the risk described in holdout reuse; it does not force the score upward in every realization. Google’s dataset-splitting lesson warns about repeated use and duplicates. When the target is performance on new cases, duplicates that supply known answers undermine that target. Information unavailable at the intended prediction time is another form of data leakage.

Match the set to deployment. Future-case evaluation must respect time and the features and labels available at each simulated cutoff. For new customers or patients, keep their groups out of training. For future predictions about known customers, group overlap may be appropriate when time and information boundaries are respected. Separate representative samples from deliberately oversampled difficult or incident cases; report stress results separately and justify any population weights. The evaluation protocol records these choices, and Train, Validation, and Test Sets without Data Leakage develops the splitting methods.

For independent binary observations with observed accuracy 90%, a normal-approximation 95% interval for a single proportion has a half-width of about 0.48 percentage points at 15,000 cases and 4.8 points at 150 cases. These are illustrative single-rate intervals, not the uncertainty of a paired model difference or a judge’s numerical rating. Compare versions using within-case differences and account for grouped cases, sampling weights, and relevant generation variability. Power and precision depend on the effect, measure, and design as well as sample size.

For generative systems

For a language model system, examples may have reference answers or criteria for what an acceptable output must contain and avoid. Prompt demonstrations and tuning examples remain useful development material; they should not be reported as an independent final test of the selected system. Keep difficult cases in a regression suite and use representative, unexposed data for confirmation. A sealed set helps control access, but repeated release verdicts can still influence the next candidate. Record that history and use fresh independent confirmation or a procedure designed for adaptive reuse. A scheduled refresh alone does not restore independence or representativeness. The Judge Preferred Longer Answers connects these roles to release and monitoring decisions.

References:

• Google for Developers, Machine Learning Crash Course: Dividing the original dataset (Creative Commons Attribution 4.0).
• scikit-learn, Cross-validation: evaluating estimator performance.