Evaluating AI Systems Before and After Release

Consider a fictional claims team’s assistant that turns long claim files into a one-page note for adjusters. For a candidate release, the team changes the model and prompt, then evaluates 150 files using a language model that scores each summary from 1 to 10. In this example, the average rises from 7.1 to 8.4 and the team deploys. Adjusters later report more rewriting, and two complaints concern summaries that omit policy exclusions. The higher judge score has not answered whether the new release is better for this task.

The investigation in this example finds that the longer summaries receive higher scores even when they omit required information. The judge comes from the candidate’s model family, but that fact alone does not establish self-preference as the cause. Thirty evaluation files have also been used as prompt demonstrations, compromising their role as unseen confirmation cases. Finally, the team has no planned comparison of editing effort, audited errors, and completed work. Edit frequency would have been useful evidence, but it too depends on how thoroughly adjusters review the output.

Offline, human, and online evidence serve different decisions

Evaluation has several overlapping dimensions. Offline and online describe where evidence is collected; human and model ratings describe who supplies a judgment. The following views serve different decisions and can be combined.

Kind	Question it answers	Strengths	Limits
Offline, on a fixed set	Is this version better than the last on known cases?	Repeatable, fast, runs on every change	Only as good as the set and the scoring
Human rating	Is this output actually good, by the standards of people who know the work?	Captures quality that no metric defines	Slow, costly, and raters disagree
Online, on real traffic	Does the system help in practice?	Measures real use and real outcomes	Arrives after exposure; outcomes are noisy and often delayed

Offline checks can support a decision to expose a candidate, human review can clarify and assess task quality, and online evidence can show what happens in use. Human ratings can be part of both offline and online evaluation. The claims team relied on a flawed automated score without the other evidence its decision required. Statistical traps such as repeated test-set selection and misleading averages also apply to generative systems; Ninety-Seven on the Test Set develops those principles.

Building the evaluation set

Public benchmarks provide evidence about their own tasks and populations. Google Cloud’s evaluation overview emphasizes the additional value of evaluation on the application’s work. Define that work’s population and time period, then sample eligible production inputs under approved data-use and access rules. Specify important segments, sampling probabilities, and the unit of independence, such as a claim or customer. Keep deliberately selected difficult cases and past incidents in a separate stress or regression suite. They help reveal known weaknesses, but their unweighted average is not an estimate of the production success rate. Report segment results and use justified population weights when estimating that rate. An incident fixture also cannot guarantee detection of every future variant.

Each example needs a basis for judging the output, a test oracle. Extraction may have a reference answer; a summary may need required facts, prohibited claims, and evidence from the source file. An assessment rubric makes those criteria explicit, but people and models can still interpret them differently. Record how reference judgments were made, review uncertain cases, and preserve unresolved disagreements rather than treating every human label as unquestionable truth.

Sample size matters in relation to the measure and design. For an illustrative sample of 150 independent binary outcomes with an observed success rate of 80%, the standard error is about 3.3 percentage points and a normal-approximation 95% interval has a half-width of about 6.4 points. That is an interval for one success proportion, not for a 1-to-10 judge score or a difference between versions.

Both versions are evaluated on the same 150 cases, so the next question is which cases changed. Moving from 120 to 126 successes is a net gain of six; the same net gain can conceal very different combinations of improvements and regressions. Paired comparisons need the within-case differences: both 6 improvements with no regressions and 30 improvements with 24 regressions change 120 successes to 126 out of 150, yet provide different evidence.

Use an appropriate paired interval or test, accounting for customer/claim clustering and relevant generation variation. Choose sample size for the effect and uncertainty that matter under an evaluation protocol, rather than reading a minimum detectable difference from a single proportion’s interval.

Contamination

An evaluation set that influences the candidate loses its role as a selection-independent estimate of performance on new cases. It can remain useful for development and regression testing. Such contamination is one form of data leakage: demonstrations or fine-tuning examples may overlap a purported final test, and repeated prompt selection can adapt to test feedback, as described in holdout reuse. Public benchmark material may also occur in pretraining data. These routes can distort evaluation without making every score a measure of memory alone. Record the overlap and the decisions it affected; deleting exposed rows afterward does not necessarily remove their influence on the selected candidate.

Separate development and regression fixtures from confirmation data. Fix the candidate, scoring rules, and decision procedure before opening the confirmation set. Restrict access and record each use, but recognize that a pass/fail release decision can still guide the next candidate even if developers never see the rows. After such adaptation, use representative data not involved in those decisions, or a statistical procedure designed for repeated adaptive use. Refreshing on a calendar is not sufficient by itself: verify independence from selection, population coverage, and duplicates. Prompt demonstrations are legitimate development material when they are not also presented as an independent final test.

How outputs are scored

Scoring methods range from exact to judgment-based, and a good evaluation uses the most exact method each criterion allows. Programmatic checks cover what can be computed: the output parses, required fields are present, cited passages exist, the stated amount matches the file. Reference comparison applies where a correct answer is known; Google’s service uses deterministic metrics when ground truth is available, though text-overlap scores reward wording more than meaning. Rubric checks turn each criterion into a pass or fail test; the same documentation describes evaluation rubrics as “similar to unit tests in software development”. Human ratings cover what no rule captures. And a model can apply rubrics or compare outputs at scale.

Model judges and their biases

A language model can grade outputs at scale as an LLM-as-a-judge. Zheng and colleagues found useful agreement with human preferences in their benchmark settings, alongside position and verbosity effects. Position preference varied with the model and prompt. They also observed some judges favoring particular models, including themselves, but explicitly said the evidence could not establish self-enhancement bias. That finding does not establish a general preference for an entire model family. Evaluate these sensitivities on the intended task; choosing a different family is an option to test, not proof of an unbiased judge.

Validate a judge on human-rated outputs that were not used to tune its prompt or rubric. Google’s judge-evaluation documentation includes confusion matrices and class-specific metrics. Inspect missed critical failures and important segments, not only aggregate agreement: a grader that calls all 100 outputs acceptable agrees on 95 if only five are unsafe, yet detects none of those five. State whether the reference is an individual rating, a consensus, or an adjudicated label. Compare with inter-annotator agreement as context, not a ceiling. If two humans each make ten non-overlapping mistakes on 100 items, they agree on 80%, while a correct judge agrees with each on 90%.

Test order effects by swapping answer positions and mapping verdicts back to the original output identities. Google’s response-flipping option swaps baseline and candidate in part of the calls; it does not establish a universal rule for classifying inconsistent verdicts. Prespecify whether those verdicts trigger review, abstention, or another aggregation rule, and report inconsistency separately from genuine ties. Repeated judge calls can reduce random variation, but can repeat the same systematic mistake. Record the rubric, prompt, sampling settings, and pinned model identifier. Pinning reduces unintended model changes; it does not make every score deterministic.

Human evaluation

Use domain experts to develop criteria, assess difficult cases, and investigate disagreement. Blinding model identities and randomizing presentation order can reduce some biases in comparisons. Low agreement is a signal to investigate: ambiguous instructions, insufficient source evidence, differing expertise, fatigue, annotation mistakes, and legitimate perspectives can all contribute. Compare reasoning and resolve cases using the task’s evidence and decision rules. After revising the rubric or judge, confirm performance on independent ratings rather than claiming validation on the same cases used to improve it.

Regression checks on every change

Changes to the model, prompt, retrieval, tools, or evaluator can alter results. Run baseline and candidate under the same evaluation rules and inspect case-level and segment-level changes, distinguishing automated flags from confirmed failures. When the judge or rubric changes, rescore stored outputs from both systems under a common evaluator; where feasible, score the same outputs under both old and new evaluators to separate measurement changes from system changes. Repeat uncertain model judgments as the protocol requires. A release policy can reject confirmed new failures on mandatory criteria even when an average improves. Shared infrastructure can run the checks, while teams and policy owners supply the relevant criteria, as described in Every Team Had Its Own Key.

From evaluation to release, and from release to monitoring

At a release gate, named owners apply prespecified conditions: required critical-case checks, acceptable deterioration by segment, validated measurement, cost and latency limits, and a decision on known failures. Define how uncertainty, multiple comparisons, and repeated candidate reviews affect that decision. No observed new failure in a sample does not prove zero production risk. Shadow testing can run copied inputs without substituting its answer for the user’s answer, but tool writes must be disabled, mocked, or isolated; copied data needs authorization and extra load must stay within resource budgets. Shadow outputs do not directly measure how users would behave if shown them. A canary deployment limits initial real exposure under explicit stop and recovery rules.

Choose a primary metric and guardrails from the intended outcome before launch. For claims work, editing frequency can help measure rework, but it is also a proxy metric: it can fall because summaries improve or because people stop checking them. Define substantial edits, the eligible-case denominator, review coverage, observation window, and missing outcomes. Pair the measure with audited omissions and incorrect claims, review/handling time, and complaints after an adequate follow-up period. To estimate the release’s causal effect, design an appropriate concurrent comparison with assignment, clustering, spillovers, and uncertainty addressed; A/B Testing and Experiment Design explains the experimental approach. A canary is not automatically a randomized experiment. A holdback group can support a longer-term comparison when its design and continued use are justified.

After launch, combine risk-appropriate immediate alerts with scheduled evaluation of sampled outputs. A weekly review alone may be too late for a severe failure. Track the sampling population, judge version, segment results, and human audit coverage alongside online metrics. Add newly discovered failures to development and regression suites; keep subsequent confirmation independent of decisions those cases informed. The Model Nobody Owned covers ownership of this loop. For document-based answers, groundedness checks relate claims to the supplied evidence, while source correctness and task usefulness remain separate questions.

What a reviewer should ask

• Which sample estimates production performance, and which separate cases test hard segments and past incidents?
• What defines a correct output for each example, and who wrote it?
• Which examples informed development, and which evidence remains independent for confirmation?
• If a model judge is used, which reference validates it, and what critical errors, segment differences, and uncertainty remain?
• What difference and uncertainty does the paired sampling design support?
• Which cases newly fail in this release, and who accepted them?
• What online metric will show whether the release helped, and when will it be read?
• How are production outputs evaluated after release, and how do new failures reach the set?

These results each have a limited interpretation. A higher average from an unvalidated judge establishes a change in that score, without establishing task improvement. A public benchmark provides evidence about its tasks, subject to contamination and sampling limits. Selected demonstrations show particular capabilities without estimating how often they succeed in the target population. None alone establishes readiness for this release.

For the fictional claims team, a stronger release process starts with adjusters specifying required facts and recording disputed judgments. A representative sample estimates task performance; a separate suite targets exclusions, amounts, and past failures. Candidate selection uses development data, followed by independent confirmation under fixed rules. Judge approval considers missed critical errors and segment performance, not whether one overall agreement number matches human agreement. The team can then stage exposure with stop rules and, where it needs a causal estimate, an appropriately assigned comparison. It should widen rollout only on evidence that audited quality and workload outcomes meet the agreed conditions, rather than on an unexplained rise in rubric scores or fall in edit frequency.

References:

• Google Cloud, Gen AI evaluation service overview (Creative Commons Attribution 4.0).
• Google Cloud, Evaluate a judge model (Creative Commons Attribution 4.0).
• Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track.
• Google Cloud, Configure a judge model.
• scikit-learn, cohen_kappa_score.
• scikit-learn, Cross-validation: evaluating estimator performance.
• statsmodels, McNemar test.