Reproducible Test Sets for AI Evaluation

Consider a hypothetical team that maintains the returns assistant from the previous articles. It has a new release of the prompt and wants to know whether the release is better. Someone pastes 30 customer questions into a script, runs both releases once, and reports that the new release answers 27 correctly and the old one 25. The team ships. A week later the share of conversations escalated to a human agent has risen, and nobody can say whether the 27-versus-25 result was ever evidence of anything.

Each part of that result can be questioned. Where did the 30 questions come from, and do they resemble the traffic that arrives? Who decided that the correct answers were correct? Was “correct” the only thing that mattered, or did one of the two wrong answers cost far more than the others? Would the same release have scored 25 or 29 if it were run again? Is a two-case difference larger than the noise in the run? And would anyone be able to rebuild the number next month, after the test file, the prompt and the model alias have each changed once?

A reproducible test set is the answer to those questions as an engineering artifact: a versioned collection of cases with agreed expected behavior, a record of how it was sampled, scorers that are tested like any other code, and a comparison procedure that accounts for noise. The returns assistant is the running example, and the problems come in the order a team meets them: what can go wrong, which cases to collect, how to score what code can check, how to score what it cannot, how to trust the scorers, and how to compare two releases. The article on framing an AI application supplies the acceptance criteria and error costs that this evaluation is meant to check, and the article on context and prompt interfaces supplies the release fingerprint and the small behavioral suite that guards the prompt itself. The programs here are plain Python 3.12 using only the standard library. All data is synthetic, written for this article, and every model in them is a stub with configured flaw rates, so the numbers show how the method behaves and say nothing about any real model.

A test set is a measuring instrument

A test set, a way of running it, and a way of scoring it together form an instrument that turns “how good is this release?” into a number. Like any instrument it has properties that can be checked separately.

• Validity: the cases resemble what the application will meet, and the score reflects what the consumer of the output cares about.
• Reliability: measuring the same release twice gives nearly the same number, which requires controlling what varies between runs.
• Sensitivity: the set is large and varied enough that a difference that matters shows up above the noise.
• Independence: the people tuning the release did not use the same cases to choose its prompt, examples or thresholds, so the score is not an echo of the tuning.

The instrument is a pipeline of stages.

test set vN  (cases + expected behavior + stratum + weight, hashed)
     |
     v
run:  release fingerprint x k runs per case  ->  stored raw outputs
     |
     v
score: deterministic checks, then judges  ->  per-case, per-run results
     |
     v
aggregate: stratum weights, per-stratum counts, intervals
     |
     v
compare: paired, same cases, against the baseline or previous release  ->  gate

The behavioral suite in the previous article, which follows the CheckList style of Ribeiro and colleagues, is the same idea at a small scale: eight cases that each exercise one rule of one prompt. It protects a prompt interface against regressions. The set built here estimates how the whole application performs on a population of requests, so it needs a sampling design, a label policy and a way to attach uncertainty to its numbers, none of which a rule-by-rule suite needs.

Task and failure taxonomy

A test set that is assembled from whatever questions are at hand tends to over-represent the easy middle of the task. A taxonomy fixes that by stating, before collecting data, which kinds of input exist and which kinds of failure the application can show. The test set is then built to cover that grid on purpose.

Two axes

The first axis is the type of input, called a stratum below: the situations the application meets that differ in how the right answer is reached. For the returns assistant these are an ordinary request inside the return window, a request on the boundary day, a final-sale item, two policy passages that conflict, a request with missing information, and a request or passage that contains an instruction aimed at the model. The second axis is the failure mode: a way the output can be wrong that someone can observe and name. Failure modes come from two places. Requirements give the first ones, because every rule in the instructions and every acceptance criterion implies a way to violate it. Observed behavior gives the rest. Reading a few dozen failing outputs from the early prototype and grouping them by what went wrong, a procedure from qualitative research called open coding, produces modes that nobody listed in advance.

A failure mode earns its place in the taxonomy when it passes three tests. An observable property distinguishes it from the other modes, so two reviewers looking at the same output would assign the same mode. It can be attached to a detector, either a deterministic check or a judge, and the taxonomy records which. And it has a cost, so that the evaluation can treat a missed security-relevant failure differently from a cosmetic one; the cost matrix from the framing article is the natural source.

Mode	Failure	Detected by	Why it matters
F1	Wrong decision on a decidable case	Check against the expected decision	Direct business error: a wrongly granted or denied return
F2	Answers when the evidence cannot support an answer	Check: expected decision is the abstention label	The costly guess
F3	Abstains when the evidence supports an answer	Check	Agent time; erodes the value of automation
F4	Bad citation: id not supplied, or deciding passage not cited	Check on the cited ids	Breaks traceability; reveals fabricated support
F5	Follows an instruction hidden in the evidence	Check: decision unchanged, forbidden content absent	Security-relevant
F6	Output violates the schema	Check	Downstream parser fails
F7	Reason states something the evidence does not support	Judge or human	Customer sees an unsupported claim; not decidable by code

Six of the seven modes can be decided by a function. Only F7 needs a judge, which is why the sections on judging come after the sections on checks: the number of cases that need human or model judgment is made as small as the task allows, because that judgment is the slow, expensive and noisy part of the evaluation.

A coverage matrix

Writing the grid down makes gaps visible. The program below records, for each case, its stratum and the failure modes it can expose, declares which modes each stratum must probe, and prints the cells that are required but empty. The cases are twelve hypothetical entries tagged by source: a log sample, hand-written synthetic text, or an incident that actually occurred in the hypothetical product.

from dataclasses import dataclass

MODES = {
    "F1": "wrong decision on a decidable case",
    "F2": "answers when the evidence cannot support an answer",
    "F3": "abstains when the evidence supports an answer",
    "F4": "bad citation: id not supplied, or deciding passage not cited",
    "F5": "follows an instruction hidden in the evidence",
    "F6": "output violates the schema",
    "F7": "reason states something the evidence does not support",
}
DETECTOR = {"F1": "check", "F2": "check", "F3": "check", "F4": "check",
            "F5": "check", "F6": "check", "F7": "judge"}

# modes that each stratum of inputs must probe at least once
REQUIRED = {
    "standard":     ["F1", "F3", "F4", "F6", "F7"],
    "boundary":     ["F1", "F4"],
    "final_sale":   ["F1", "F4", "F7"],
    "conflict":     ["F1", "F4", "F7"],
    "missing_info": ["F2", "F3"],
    "injection":    ["F1", "F5"],
}

@dataclass(frozen=True)
class Case:
    id: str
    stratum: str
    modes: tuple      # failure modes this case can expose
    source: str       # log | synthetic | incident

CASES = [
    Case("c01", "standard", ("F1", "F6"), "log"),
    Case("c02", "standard", ("F1", "F4"), "log"),
    Case("c03", "standard", ("F3", "F6"), "synthetic"),
    Case("c04", "boundary", ("F1",), "synthetic"),
    Case("c05", "boundary", ("F1",), "synthetic"),
    Case("c06", "final_sale", ("F1", "F4"), "log"),
    Case("c07", "final_sale", ("F1",), "incident"),
    Case("c08", "conflict", ("F1",), "incident"),
    Case("c09", "missing_info", ("F2",), "synthetic"),
    Case("c10", "missing_info", ("F2", "F3"), "synthetic"),
    Case("c11", "injection", ("F5", "F1"), "incident"),
    Case("c12", "injection", ("F5",), "synthetic"),
]

modes = sorted(MODES)
print("coverage: cases per (stratum, failure mode); '.' = required but uncovered")
print(f"{'stratum':13s}" + "".join(f"{m:>4s}" for m in modes) + "   n")
gaps = []
for s, req in REQUIRED.items():
    row = 
    cells = []
    for m in modes:
        k = sum(m in c.modes for c in row)
        if m in req and k == 0:
            cells.append("   ."); gaps.append((s, m))
        else:
            cells.append(f"{k:4d}" if k else "   -")
    print(f"{s:13s}" + "".join(cells) + f"{len(row):4d}")
print("\nuncovered required cells:")
for s, m in gaps:
    print(f"  {s:12s} {m} ({MODES[m]}; detected by {DETECTOR[m]})")
by_src = {}
for c in CASES:
    by_src = by_src.get(c.source, 0) + 1
print("\ncases by source:", by_src)

Its output:

coverage: cases per (stratum, failure mode); '.' = required but uncovered
stratum        F1  F2  F3  F4  F5  F6  F7   n
standard        2   -   1   1   -   2   .   3
boundary        2   -   -   .   -   -   -   2
final_sale      2   -   -   1   -   -   .   2
conflict        1   -   -   .   -   -   .   1
missing_info    -   2   1   -   -   -   -   2
injection       1   -   -   -   2   -   -   2

uncovered required cells:
  standard     F7 (reason states something the evidence does not support; detected by judge)
  boundary     F4 (bad citation: id not supplied, or deciding passage not cited; detected by check)
  final_sale   F7 (reason states something the evidence does not support; detected by judge)
  conflict     F4 (bad citation: id not supplied, or deciding passage not cited; detected by check)
  conflict     F7 (reason states something the evidence does not support; detected by judge)

cases by source: {'log': 3, 'synthetic': 6, 'incident': 3}

Read the table as a work list. A number is the count of cases that probe that combination; a dash means the mode does not apply to the stratum or no one required it; a dot is a hole. The twelve cases leave five required cells empty: nothing probes citation behavior on boundary and conflicting passages, and nothing in the three strata where the reason is likely to repeat policy text probes F7. A team that sees the holes can decide to fill them, to drop the requirement, or to record that the claim “citations are correct” is currently untested on those strata. All three are better than finding the hole in production. The matrix also shows the source mix. Three incident cases are valuable because they came from real failures, but they are a biased sample of what can go wrong, so they belong to a separate slice, as the next section explains.

The taxonomy is a living document. A new failure observed in production becomes a new mode or a new case under an existing one, and the matrix is rerun. Failures that cannot be told apart by an observable property should be merged, and a mode that no detector can reach should be recorded as a known blind spot instead of silently dropped.

Representative data

“Representative” has a precise meaning for an estimate: the cases are drawn so that the weighted pass rate of the test set approximates the pass rate on the traffic the application will receive. That meaning is different from “interesting” and from “covers every mode”, and a test set usually has to serve both purposes. The way to serve both without confusing them is to keep three kinds of cases in separate, named slices.

Slice	How cases are chosen	Question it answers	How to report it
Representative	Probability sample of real or realistic requests, stratified, with known weights	What pass rate do we expect on traffic?	Weighted rate with an interval
Stress	Written or generated to probe a taxonomy cell, including adversarial text	Does this mode ever fail, and how often under pressure?	Per-mode rate; never averaged into the headline
Regression	Cases from incidents and fixed bugs	Did a known failure return?	Pass or fail per case; any new failure is a finding

Pooling the three into one average is the most common way a test set misleads. The pooled number depends on how many stress and incident cases someone happened to add, so it is a statement about the test set’s composition and not about the application. Keeping the slices apart lets each one answer its own question.

Sampling the representative slice

The representative slice starts from a frame, the list that requests are drawn from, typically a log of recent traffic. The frame determines what the sample can represent: a log that excludes requests handled in another channel represents only the channels it covers. The glossary entry on statistical sampling explains the distinction between the population, the frame and the selection method. A log that holds real customer text also holds personal data, so what is stored in the test set, who can read it, and what is scrubbed first are decisions to make before sampling; the article on privacy and output controls covers the controls.

Simple random sampling from the frame gives an unbiased estimate, but it leaves rare strata with few cases. Suppose injection attempts are 3% of traffic. A random sample of 300 requests contains about nine of them, and the pass rate on injection, which is the stratum whose failures are most expensive, is estimated from nine cases. Stratified sampling fixes the number of cases per stratum in advance and draws randomly inside each stratum. Let \(W_h\) be the share of traffic in stratum \(h\), \(n_h\) the number of sampled cases in it, and \(\hat p_h\) the observed pass rate among them. The traffic-weighted estimate of the overall pass rate is

\[ \hat p = \sum_h W_h\, \hat p_h , \qquad \widehat{\mathrm{Var}}(\hat p) \approx \sum_h W_h^2\, \frac{\hat p_h (1-\hat p_h)}{n_h}. \]

In words: estimate each stratum separately, then combine the estimates with the share of traffic each stratum represents. The variance formula adds the sampling variance of each stratum, scaled by its weight squared, and ignores the finite-population correction. It shows the design choice. A stratum with a large weight and a pass rate near 0.5 deserves many cases, and a stratum with a tiny weight needs few if all one wants is the overall rate. But the same stratum may need many cases if its own rate matters, which is the case for injection. Allocation is therefore a compromise between precision of the headline number and precision of the slices that carry the most risk. Equal-allocation designs maximize per-stratum precision at the price of a wider overall interval; proportional allocation does the reverse. A middle path sets a floor of cases per stratum and distributes the rest proportionally.

The program below builds a hypothetical population of 20,000 logged requests in six strata, each with its own pass rate for a release under test, so that the true overall pass rate is known. It then draws 2,000 repeated samples of 300 labeled cases under three designs and reports how the estimates vary. The traffic shares and pass rates are invented for this example.

import random, statistics, math

# hypothetical traffic: (stratum, share of traffic, pass rate of the release under test)
STRATA = [("standard", .62, .97), ("late", .12, .95), ("final_sale", .08, .90),
          ("missing_info", .10, .85), ("conflict", .05, .80), ("injection", .03, .70)]
N, n, REPS = 20000, 300, 2000

rng = random.Random(7)
pop = []
for name, share, rate in STRATA:
    pop += [(name, rng.random() < rate) for _ in range(round(N * share))]
by = {name: [p for s, p in pop if s == name] for name, _, _ in STRATA}
W = {name: len(v) / len(pop) for name, v in by.items()}
truth = sum(p for _, p in pop) / len(pop)
truth_inj = sum(by["injection"]) / len(by["injection"])
print(f"population {len(pop)} requests; overall pass rate {truth:.4f}; injection {truth_inj:.3f}")

def alloc_prop():
    return {k: round(n * W[k]) for k in W}

def alloc_floor(floor=30):
    rest = n - floor * len(W)
    return {k: floor + round(rest * W[k]) for k in W}

def srs(r):
    s = r.sample(pop, n)
    est = sum(p for _, p in s) / n
    inj = [p for st, p in s if st == "injection"]
    return est, len(inj), (sum(inj) / len(inj) if inj else None)

def stratified(r, alloc):
    est, inj_n, inj_rate = 0.0, alloc["injection"], None
    for k, nh in alloc.items():
        s = r.sample(by[k], nh)
        ph = sum(s) / nh
        est += W[k] * ph                      # weight each stratum by its traffic share
        if k == "injection":
            inj_rate = ph
    return est, inj_n, inj_rate

def report(name, fn):
    r = random.Random(11)
    ests, ns, inj = [], [], []
    for _ in range(REPS):
        e, k, ir = fn(r)
        ests.append(e); ns.append(k)
        if ir is not None: inj.append(ir)
    print(f"{name:22s} mean {statistics.fmean(ests):.4f}  sd {statistics.stdev(ests):.4f}"
          f"  | injection cases: median {statistics.median(ns):4.0f}, min {min(ns):3d}"
          f"  | sd of injection rate {statistics.stdev(inj):.3f} (undefined in {REPS-len(inj)} runs)")

print(f"\nn = {n} labeled cases per sample, {REPS} repeated samples")
report("simple random", srs)
a = alloc_prop();  print("proportional allocation:", a)
report("stratified, proportional", lambda r: stratified(r, a))
a = alloc_floor(); print("floor-30 allocation:   ", a)
report("stratified, floor of 30", lambda r: stratified(r, a))

population 20000 requests; overall pass rate 0.9339; injection 0.725

n = 300 labeled cases per sample, 2000 repeated samples
simple random          mean 0.9338  sd 0.0140  | injection cases: median    9, min   1  | sd of injection rate 0.160 (undefined in 0 runs)
proportional allocation: {'standard': 186, 'late': 36, 'final_sale': 24, 'missing_info': 30, 'conflict': 15, 'injection': 9}
stratified, proportional mean 0.9339  sd 0.0137  | injection cases: median    9, min   9  | sd of injection rate 0.146 (undefined in 0 runs)
floor-30 allocation:    {'standard': 104, 'late': 44, 'final_sale': 40, 'missing_info': 42, 'conflict': 36, 'injection': 34}
stratified, floor of 30 mean 0.9342  sd 0.0136  | injection cases: median   34, min  34  | sd of injection rate 0.077 (undefined in 0 runs)

All three designs are unbiased: the means of the estimates sit at the true value of 0.9339 to within 0.0004. The headline precision is nearly the same, with a standard deviation of about 0.014 for each design, because the overall rate is dominated by the standard stratum, which all designs cover well. The designs differ where it matters for risk. Under simple random sampling the median number of injection cases is 9 and the minimum is 1, and the injection pass rate, whose true value is 0.725, varies with a standard deviation of 0.160 from sample to sample. Proportional stratification pins the count at 9 and gains little. The floor-of-30 design takes 34 injection cases every time and halves the standard deviation of the injection estimate to 0.077, at no visible cost to the headline number. The cost is that the data must carry its weight: a floor design makes the raw pass rate over all 300 cases wrong, because the rare strata are overrepresented, and only the weighted estimate recovers the traffic value. Every case therefore stores its stratum, and the report states both the weighted rate and the per-stratum counts.

Expected behavior and labels

Each case needs an expected behavior that a scorer can compare to. For a classification-like field such as the decision, the expected value is a label. For a generated explanation there is no single right string, so the expected behavior is a set of properties: facts that must appear, facts that must not, and the evidence ids that must be cited. Properties are checkable by code where they are structured and by a judge otherwise.

Labels come from people, who disagree. The label policy has to say who writes them, from what written guideline, and what happens on disagreement. A workable pattern is that one trained person labels every case from the guideline, a second person labels a random subset, disagreements are adjudicated by a third and used to sharpen the guideline, and the agreement rate on the double-labeled subset is reported with the test set. Noisy labels set a ceiling on what the set can show: if a few percent of expected decisions are wrong, a release that is better than the labels cannot be measured as such. The section on human judging below shows how to measure agreement, and the glossary entry on inter-annotator agreement covers the statistic in more depth.

Splits and leakage

A test set loses its meaning when information from it reaches the people or procedures that tune the release. Three roles need separate cases. The development slice is what the team looks at while writing and changing prompts, so the team can read its failures. The regression slice holds fixed cases that every candidate must keep passing. The confirmation slice is touched only to decide whether a candidate ships; the glossary entries on the evaluation set and holdout reuse explain why even a confirmation slice stops being independent when its results steer the next candidate.

Two sources of leakage are specific to this kind of application. The first is that one underlying request often appears several times in a log, as a resubmission, a follow-up, or a copy from the same customer, so a random split by case puts versions of one request on both sides. The second is that few-shot examples in the prompt are cases: an example that is also in the test set makes the test measure memorization. The same logic covers data that is public. Jacovi and colleagues note that test data published in plain text on the web can end up in the training data of the models being evaluated, and propose measures such as encrypting released test data. A private test set that was never published avoids the problem, though text copied from a public source into it does not.

The program below builds 240 synthetic cases from 60 underlying requests, each resubmitted four times with light edits such as a greeting or a thank-you. It detects near-duplicates with the Jaccard similarity of word trigrams, \(|A \cap B| / |A \cup B|\), where \(A\) and \(B\) are the sets of three-word sequences in two texts, and compares a random split by case with a split by underlying request.

def norm(t):
    return re.sub(r"[^a-z0-9 ]", " ", t.lower()).split()

def shingles(t, k=3):
    w = norm(t)
    return {tuple(w[i:i + k]) for i in range(len(w) - k + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

def leaked(dev, test, thr=0.6):
    sd = [shingles(c["text"]) for c in dev]
    return sum(any(jaccard(shingles(c["text"]), s) >= thr for s in sd) for c in test)

cases = make_cases()
rng = random.Random(3)

# split A: random by case
idx = list(range(len(cases))); rng.shuffle(idx)
cut = int(0.7 * len(cases))
dev_a = [cases[i] for i in idx[:cut]]; test_a = [cases[i] for i in idx[cut:]]

# split B: random by group
groups = sorted({c["group"] for c in cases}); rng.shuffle(groups)
gcut = int(0.7 * len(groups)); dev_groups = set(groups[:gcut])
dev_b =  in dev_groups]
test_b =  not in dev_groups]

print(f"{len(cases)} cases from {len(groups)} underlying requests")
for name, dev, test in [("random split by case", dev_a, test_a), ("random split by group", dev_b, test_b)]:
    print(f"{name:22s} dev {len(dev):3d}  test {len(test):3d}  "
          f"test cases with a near-duplicate in dev: {leaked(dev, test):3d}")
a, b = cases[0]["text"], cases[1]["text"]       # same request, resubmitted with a greeting
c = cases[4]["text"]                              # a different request
print(f"jaccard(resubmission) {jaccard(shingles(a), shingles(b)):.2f}; jaccard(different request) {jaccard(shingles(a), shingles(c)):.2f}")

240 cases from 60 underlying requests
random split by case   dev 168  test  72  test cases with a near-duplicate in dev:  68
random split by group  dev 168  test  72  test cases with a near-duplicate in dev:   0
jaccard(resubmission) 0.93; jaccard(different request) 0.16

The Jaccard similarity of a resubmission and its original is 0.93, and that of two different requests is 0.16, so a threshold of 0.6 separates them for this data. With the random split by case, 68 of the 72 test cases have a near-duplicate in the development slice, so a release tuned on development data would be partly tested on its own examples. Splitting by request puts every version of a request on one side and leaves no near-duplicates across the split. The lexical measure only catches texts that share wording. Paraphrases that share meaning and little wording pass through, so a production pipeline adds an embedding similarity check and, where the log carries a customer or conversation id, splits on that id. The article on train, validation and test sets without data leakage develops the grouped and temporal splits for classical models, and the same ideas apply here.

How many cases

The number of cases sets the width of the interval around every rate. For a pass rate near \(p\) measured on \(n\) independent cases, the normal approximation gives a 95% half-width of about \(1.96\sqrt{p(1-p)/n}\). To estimate a pass rate near 0.90 to within plus or minus 0.05 takes about \(1.96^2 \times 0.9 \times 0.1 / 0.05^2 \approx 139\) cases, and a rate near 0.97 to within 0.02 takes about 280. Near the edges the Wilson interval used in the framing article behaves better than this approximation, but the arithmetic shows the scale. The size that matters is the size of the slice that carries a requirement, so a stratum with a recall target needs its own cases, not a share of the total. Detecting a difference between two releases needs more cases than estimating one rate, and the section on regression returns to it.

Making a run reproducible

Reproducibility here means that someone else, or the same team later, can recompute a reported number and can explain any difference between two numbers. Several things vary between evaluations, and each needs a control.

What varies	Control
The cases and labels	Version the test set; hash a canonical serialization; document its sampling, labeling and known gaps
The release	Use the release fingerprint from the prompt article: prompt, template, examples, exact model identifier, decoding parameters
Randomness inside a run	Run each case k times and report the per-case pass rate, not one sample; record seeds where the provider accepts them
Provider-side variation	Assume outputs can differ for identical input even at temperature zero, so store outputs and measure the variation
The scorers	Version the checkers and the judge prompt; score as a pure function of stored outputs
The aggregation	Keep weights and the formula in code with the results

Biderman and colleagues, who maintain the Language Model Evaluation Harness, identify sensitivity to evaluation setup, difficulty comparing results across setups, and lack of reproducibility and transparency as recurring problems in language model evaluation. The controls above address the same problems at application scale.

Two practices make most of the difference. The first is to separate generating outputs from scoring them. Calling the model is slow, costs money and cannot be replayed exactly, so the harness writes every raw output to a run record together with the test set hash, the release fingerprint, the case id and the run index. Scoring then reads the record and nothing else, so it can be repeated, changed and audited without another model call. The second is to document the test set the way a dataset is documented. Gebru and colleagues propose a datasheet for each dataset that records why it was created, what it contains, how it was collected and labeled, and what it should not be used for. For a test set, the useful entries are the frame and sampling design, the stratum weights, the label policy and its measured agreement, the slices and their purposes, and the strata that are known to be thin.

The harness below implements the first practice and shows what the second protects. The test set is ten cases, stored as dictionaries. A canonical serialization (sorted keys, fixed separators, ASCII) gives each case a hash, and the hash of the sorted case hashes identifies the set. The stub release flips decisions, cites an id that was not supplied, returns malformed output or obeys an injected passage with configured probabilities. The excerpt shows the hashing; the checks follow in the next section.

import hashlib, json, random, zlib

# ---------- test set: canonical serialization and a content hash ----------
def canon(obj):
    return json.dumps(obj, sort_keys=True, ensure_ascii=True, separators=(",", ":"))

def sha(s):
    return hashlib.sha256(s.encode("utf-8")).hexdigest()

def testset_hash(cases):
    return sha("\n".join(sorted(sha(canon(c)) for c in cases)))[:12]

The next excerpt is the part that writes and reads run records. Each stored line holds the release fingerprint, the test set hash, the case id, the run index and the raw output string.

# ---------- run records and scoring ----------
def run_release(rel, cases, k, path):
    th = testset_hash(cases)
    with open(path, "w") as f:
        for c in cases:
            for i in range(k):
                f.write(canon({"release": rel.fingerprint(), "name": rel.name, "testset": th,
                               "case": c["id"], "run": i, "output": rel.generate(c, i)}) + "\n")

def score(path, cases, **kw):
    byid = {c["id"]: c for c in cases}
    tot = {k: 0 for k in CHECKS}; n = 0
    for line in open(path):
        rec = json.loads(line); n += 1
        for k, fn in CHECKS.items():
            tot[k] += fn(rec["output"], byid[rec["case"]], **kw)
    return n, {k: round(v / n, 3) for k, v in tot.items()}

K = 20
rel = Release("release-a")
run_release(rel, CASES, K, "runs_a.jsonl")
print("test set hash:", testset_hash(CASES), " release fingerprint:", rel.fingerprint())
n, s1 = score("runs_a.jsonl", CASES)
n, s2 = score("runs_a.jsonl", CASES)
print(f"{n} stored outputs; strict scoring        {s1}")
print("rescored from the same file, identical:", s1 == s2)
n, s3 = score("runs_a.jsonl", CASES, accept_fences=True)
print(f"scoring with fenced JSON accepted {s3}   (no model call)")

fixed = [dict(c) for c in CASES]
fixed[2] = dict(fixed[2], decision="not_eligible")        # a label edit: day 30 treated as outside the window
print("test set hash after editing one label:", testset_hash(fixed))
n, s4 = score("runs_a.jsonl", fixed)
print(f"same outputs, edited label        {s4}")

checker tests: 7 cases, all as expected
test set hash: 755fc7f40dc9  release fingerprint: 4f204b87
200 stored outputs; strict scoring        {'schema': 0.97, 'decision': 0.93, 'citations': 0.965}
rescored from the same file, identical: True
scoring with fenced JSON accepted {'schema': 1.0, 'decision': 0.96, 'citations': 0.995}   (no model call)
test set hash after editing one label: 5a6ddada3892
same outputs, edited label        {'schema': 0.97, 'decision': 0.835, 'citations': 0.965}

Twenty runs of each of the ten cases give 200 stored outputs. Scoring the file twice yields identical numbers, which is the property that makes a reported figure auditable. The third score line shows what changes when a scorer changes. Accepting JSON wrapped in a code fence raises the schema pass rate from 0.97 to 1.0 and the decision pass rate from 0.93 to 0.96, and no model call was needed to find out. This is also a reason to treat the scorer as part of the release record: the same outputs received two different scores because an output-normalization policy changed. The last two lines show the effect of a data change. Editing one label alters the test set hash, and the decision pass rate on the same outputs falls from 0.93 to 0.835 because the stub, which answers the original label, now disagrees on the edited case in every run. A hash that changes whenever a label does turns “the test set was updated” into a visible event, so that numbers computed against different versions are not compared by accident.

None of this makes a hosted model deterministic. The record preserves what the model returned on the day, and the repeated runs show how much it varies, but a later rerun may return different text. Reproducibility of the evaluation therefore means that the stored outputs produce the same score and that a fresh run is expected to fall within the measured variation.

Deterministic checks

A deterministic check is a function from an output and a case to pass or fail that involves no model. It is the cheapest, fastest and most repeatable kind of scorer, and the evaluation should use it for everything it can decide. The decision, the cited ids and the schema in the returns assistant are all within its reach.

Check type	What it decides	Typical use	Blind spot
Schema and enumeration	The output parses and every field has an allowed type and value	Output contract	A valid structure with wrong content
Exact match after normalization	A field equals the expected value after a stated normalization	Labels, ids, short answers	The normalization rules are a policy choice
Set comparison	Precision and recall of a returned set against an expected set	Cited ids, extracted entities	Order and partial credit need explicit rules
Numeric tolerance	A computed value is within an absolute or relative bound	Amounts, dates, counts	Wrong units that happen to fit the bound
Containment and pattern	Required text is present; forbidden text is absent	Required disclosures, leaked secrets, injected phrases	Paraphrases of the forbidden content
Cross-field consistency	Fields agree with each other and with the input	Decision versus reason; cited ids versus supplied ids	Consistent but wrong
Execution	Generated code or queries run and produce the expected result	Code and SQL generation	Tests that miss the behavior

A check is only as good as its specification, and the specification contains decisions that are easy to overlook. Whether “Eligible” counts as the same answer as “eligible” is a decision. Whether fenced JSON is accepted is a decision. Whether a citation of an extra supplied passage is a failure or a harmless addition is a decision. Each should be made deliberately, in the contract that the application already enforces in production, and then encoded once. The harness above is strict: the output must parse as a JSON object with exactly the three fields, the decision must be one of three strings, and the cited ids must contain every required id and nothing that was not supplied.

# ---------- deterministic checks: pure functions of (output, case) ----------
def parse(out, accept_fences=False):
    if accept_fences and out.startswith("```"):
        out = out.strip("`").removeprefix("json").strip()
    try:
        d = json.loads(out)
    except json.JSONDecodeError:
        return None
    ok = (isinstance(d, dict) and set(d) == {"decision", "reason", "policy_ids"}
          and d["decision"] in ("eligible", "not_eligible", "needs_human")
          and isinstance(d["reason"], str) and isinstance(d["policy_ids"], list)
          and all(isinstance(x, str) for x in d["policy_ids"]))
    return d if ok else None

def check_schema(out, case, **kw):
    return parse(out, **kw) is not None

def check_decision(out, case, **kw):
    d = parse(out, **kw)
    return d is not None and d["decision"] == case["decision"]

def check_citations(out, case, **kw):
    d = parse(out, **kw)
    if d is None:
        return False
    supplied = {e["id"] for e in case["ev"]}
    cited = set(d["policy_ids"])
    return cited <= supplied and set(case["cite"]) <= cited

CHECKS = {"schema": check_schema, "decision": check_decision, "citations": check_citations}

# ---------- tests for the checkers themselves ----------
T = CASES[0]
GOOD = '{"decision":"eligible","reason":"x","policy_ids":["R-12"]}'
checker_tests = [
    (GOOD, True, True, True),
    (GOOD.replace('"eligible"', '"Eligible"'), False, False, False),
    (GOOD.replace("]", ',"R-99"]'), True, True, False),           # cites an id never supplied
    (GOOD.replace('["R-12"]', '[]'), True, True, False),           # deciding passage not cited
    (GOOD.replace("}", ',"extra":1}'), False, False, False),      # extra field breaks the contract
    ("```json\n" + GOOD + "\n```", False, False, False),          # fenced JSON: strict mode rejects
    ("not json", False, False, False),
]
for out, s, d, c in checker_tests:
    assert (check_schema(out, T), check_decision(out, T), check_citations(out, T)) == (s, d, c), out
print(f"checker tests: {len(checker_tests)} cases, all as expected")

The checks return a boolean, but the run record keeps the raw output, so a failing case can be traced to a failure mode. The tests at the bottom matter as much as the checks. A checker is code with a specification and its own defects, and a defect in a checker shifts every number computed with it. Each test case here pins one decision: capitalization fails the schema, an extra field fails it, an unsupplied citation fails the citation check but leaves the decision check unaffected, an empty citation list fails because the deciding passage is missing, and fenced JSON fails because the contract says one JSON object. The program asserts all of them and prints the count, so a change to a check that alters one of these decisions fails before it changes a score.

Passing deterministic checks establishes that the output has the properties that were checked. It does not establish that the output is good. A valid decision that cites the right passage can still carry a reason that misstates the policy, which is why F7 exists, and a check that is too lenient under-reports failures while one that is too strict reports failures that a customer would not notice. Both kinds of error are visible in the results of an audit, which reads a sample of passes and a sample of failures and asks whether the check agreed with a careful reader. Ordering helps cost as well: run the cheap checks first, send only outputs that pass them to a judge, and count outputs that fail the contract as failures without spending a judgment on them.

Human and model judging

Some properties cannot be reduced to a function. Whether a reply is polite, whether a summary omits a material fact, and whether an explanation is supported by the evidence in front of the model are judgments. They are made by people, by a language model prompted as a grader, or by both, and in each case the judgment is a measurement with its own error.

Writing the rubric

The quality of the judgment depends first on the rubric. A rubric that asks “Is this a good answer, 1 to 5?” gets answers that depend on the rater’s habits. A better rubric splits the question into binary criteria that each name an observable property, supplies the evidence the rater needs, and includes a few anchor examples of each verdict. For F7 in the returns assistant, the criterion could be: “Every factual claim in the reason appears in the supplied passages. Answer supported, unsupported or cannot tell.” The cannot-tell option matters, because forcing a verdict on an ambiguous case produces noise that looks like disagreement. Procedures also matter. Raters see outputs without the identity of the release that produced them, in randomized order, and the rater id is stored with every judgment so that a drifting rater can be detected.

Measuring agreement between people

Two raters who follow the same rubric still disagree on some cases, and the amount of disagreement bounds how much a judgment can be trusted. The simplest summary is the raw agreement, the share of cases that get the same label. It misleads when one label is common. Cohen’s kappa corrects for the agreement expected by chance. With observed agreement \(p_o\), and \(p_e\) the agreement expected if each rater labeled independently with their own marginal label frequencies,

\[ \kappa = \frac{p_o – p_e}{1 – p_e}, \qquad p_e = p_A p_B + (1-p_A)(1-p_B), \]

where \(p_A\) and \(p_B\) are the shares of cases that raters A and B label unacceptable. In words: kappa is the fraction of the possible improvement over chance that the raters achieved. The statistic is implemented in scikit-learn as cohen_kappa_score. Kappa equals 1 for perfect agreement and 0 when agreement is what independent labeling would produce, and it is undefined when \(p_e = 1\), which happens when both raters always use the same single label. The first block of the program below simulates two raters who each flip the true label 4% of the time, once for a task where 40% of replies are unacceptable and once for a task where 5% are.

import random, statistics

# ---------- A. agreement between two human raters (synthetic ratings) ----------
def kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    pa, pb = sum(a) / n, sum(b) / n                   # share of label 1 for each rater
    pe = pa * pb + (1 - pa) * (1 - pb)
    return po, pe, (po - pe) / (1 - pe)

def raters(prev_bad, n, err, seed):
    r = random.Random(seed)
    truth = [r.random() < prev_bad for _ in range(n)]            # 1 = unacceptable reply
    flip = lambda t: (not t) if r.random() < err else t
    return [int(flip(t)) for t in truth], [int(flip(t)) for t in truth]

for prev in (0.40, 0.05):
    a, b = raters(prev, 2000, 0.04, seed=1)
    po, pe, k = kappa(a, b)
    print(f"unacceptable prevalence {prev:.2f}: raw agreement {po:.3f}, chance agreement {pe:.3f}, kappa {k:.3f}")

# ---------- B. a pairwise judge with position and length bias (stub) ----------
rng = random.Random(5)
N = 4000
def judge_prefers_first(q1, len1, q2, len2):
    s1 = q1 + 0.30 + 0.010 * (len1 - 80) + rng.gauss(0, 1)       # +0.30: bonus for being shown first
    s2 = q2 + 0.010 * (len2 - 80) + rng.gauss(0, 1)
    return s1 > s2

def trial(len_b):
    qa, qb = rng.gauss(0, 1), rng.gauss(0, 1)        # equal quality distributions for A and B
    la, lb = 60, len_b
    ab = judge_prefers_first(qa, la, qb, lb)          # A shown first
    ba = judge_prefers_first(qb, lb, qa, la)          # B shown first
    return (not ab), ba                               # "B wins" in each order

for len_b in (60, 110):
    res = [trial(len_b) for _ in range(N)]
    b_second = sum(x for x, _ in res) / N
    b_first = sum(y for _, y in res) / N
    both_b = sum(x and y for x, y in res) / N
    both_a = sum((not x) and (not y) for x, y in res) / N
    print(f"len(B)={len_b:3d}: B wins when shown second {b_second:.3f}, first {b_first:.3f}; "
          f"swap-consistent B {both_b:.3f}, A {both_a:.3f}, inconsistent (tie) {1-both_b-both_a:.3f}")

The second block simulates a pairwise model judge, described in the next subsection. The file prints:

unacceptable prevalence 0.40: raw agreement 0.922, chance agreement 0.517, kappa 0.838
unacceptable prevalence 0.05: raw agreement 0.922, chance agreement 0.835, kappa 0.526
len(B)= 60: B wins when shown second 0.441, first 0.566; swap-consistent B 0.331, A 0.324, inconsistent (tie) 0.344
len(B)=110: B wins when shown second 0.533, first 0.663; swap-consistent B 0.423, A 0.227, inconsistent (tie) 0.350

true rate 0.119 | judge flag rate 0.147 | validation: sens 0.83 spec 0.92
corrected estimate 0.090
95% bootstrap interval [0.045, 0.135]
humans alone on 300 replies: 0.160
denominator se+sp-1 = 0.754 (near 0 means the judge carries no signal)
true rate            mean 0.120  sd 0.006
raw judge flag rate  mean 0.157  sd 0.006
corrected estimate   mean 0.121  sd 0.022
humans only (300)    mean 0.120  sd 0.019

The first two lines show why raw agreement cannot stand alone. Both raters have the same error rate in both runs and the same raw agreement of 0.922. At 40% prevalence the chance agreement is 0.517 and kappa is 0.838. At 5% prevalence the chance agreement is 0.835, so most of the observed agreement is what two raters who mostly say “acceptable” would reach anyway, and kappa falls to 0.526. A kappa that is low because the label is rare does not mean the raters are worse, and a high raw agreement does not mean they are good, so a label policy reports raw agreement, kappa and the label counts together. Agreement also measures consistency, not correctness: two raters can agree on a wrong label, and a rater who is wrong less often than another can still show low agreement with them.

Model judges

A language model prompted to grade outputs scales to far more cases than people can label, and its verdicts are repeatable in the sense that the prompt and settings can be recorded. They are not free of systematic error. Zheng and colleagues, who studied model judges on a chat benchmark, report position and verbosity biases and limited reasoning ability. They also examine possible self-enhancement bias, but their data do not establish whether it is present. They also report that strong judges agreed with human preferences at over 80% on the tasks they studied, about the same as humans agree with each other. The glossary entry on LLM-as-a-judge summarizes the forms and the checks. Two forms are common. A pointwise judge reads one output and returns a verdict against a rubric. A pairwise judge reads two outputs for the same input and picks one, which is easier for the judge and gives a relative result.

A pairwise judge shows its biases directly. The second block of the program simulates 4,000 comparisons between releases A and B in which both draw quality from the same distribution, so an unbiased judge would prefer each half the time. The stub judge adds a bonus to whichever output is shown first and a small bonus per word of length, plus noise. Both settings are configured. In the first run B has the same length as A, 60 words, and in the second B has 110 words.

With equal lengths, B wins 0.441 of the comparisons when it is shown second and 0.566 when it is shown first. Averaged over both orders the result is 0.50, as it should be, but the single-order result swings by 12 points depending on the order. Presenting every pair in both orders and counting a result only when the two verdicts agree removes the position effect: B wins both orders in 0.331 of the pairs and A in 0.324, and 0.344 of the pairs are inconsistent and are treated as ties. With the longer B, B wins 0.533 in second position and 0.663 in first, and in the swap-consistent comparison B wins 0.423 against 0.227 for A. The swap removed the position effect and left the length effect, because longer output is favored in both orders. The true quality is identical, so the 0.423 versus 0.227 is a measurement artifact, and a team that reads it as “B is better” has been measured by its own judge. Defenses against verbosity bias include a rubric that says length is not a criterion, comparing outputs of similar length, and including length-matched control pairs in the validation described next.

Validating a judge

A judge is validated against human labels before its scores are used, and the validation data must not be the data that was used to write or tune the judge’s prompt. Otherwise the agreement measures how well the prompt was fitted to those labels. The validation set is a sample of outputs labeled by people under the same rubric, with the labeling quality measured as above. The judge’s verdicts are compared with the human labels per failure type and per stratum, using error rates in each direction and not a single agreement figure. For a judge that flags unsupported reasons, the two rates are the sensitivity, the share of truly unsupported reasons that the judge flags, and the specificity, the share of supported reasons that it passes. A judge that passes everything has perfect specificity and zero sensitivity, and it can still show 95% agreement when 95% of the replies are fine, which is the failure a single agreement number hides.

Calibration and regression

Calibrating a judge’s estimate

A judge with imperfect sensitivity and specificity gives a biased estimate of the rate it is supposed to measure. If the true rate of unsupported reasons is \(\pi\), the judge flags a share

\[ q = \pi\, s_e + (1-\pi)(1-s_p), \]

where \(s_e\) is the sensitivity and \(s_p\) the specificity. The first term counts the truly unsupported replies that the judge catches, and the second counts the supported replies that it wrongly flags. Solving for \(\pi\) gives the correction

\[ \hat\pi = \frac{q + s_p – 1}{s_e + s_p – 1}, \]

which epidemiologists know as the Rogan-Gladen estimator for prevalence from an imperfect test. Its inputs are the judge’s flag rate \(q\) on the full set of outputs and estimates of \(s_e\) and \(s_p\) from the human-labeled validation sample. The denominator shows the limit of the method. When \(s_e + s_p\) is close to 1, the judge flags independently of the truth, the denominator is near zero, and the correction amplifies noise without bound. The method is also valid only if the error rates measured on the validation sample hold for the outputs being scored, so a validation sample that omits the hard cases will overstate them.

The third block of the program simulates 3,000 replies with a true unsupported rate near 12%, a stub judge with sensitivity 0.80 and specificity 0.93 as configured settings, and a validation sample of 300 replies that humans also label. The fourth block repeats the whole procedure 1,000 times with fresh data to separate bias from luck.

# ---------- C. validating a pointwise judge and correcting its estimate ----------
r = random.Random(21)
SENS, SPEC = 0.80, 0.93           # configured judge behavior: P(flag | bad), P(no flag | ok)
def judged(bad): return (r.random() < SENS) if bad else (r.random() > SPEC)
M, V = 3000, 300
truth = [r.random() < 0.12 for _ in range(M)]                     # 1 = reply has an unsupported claim
flags = [judged(t) for t in truth]
val = r.sample(range(M), V)                                       # replies that humans also label
pairs = [(truth[i], flags[i]) for i in val]

def estimate(pairs, flag_rate):
    bad = [f for t, f in pairs if t]; ok = [f for t, f in pairs if not t]
    se, sp = sum(bad) / len(bad), 1 - sum(ok) / len(ok)
    return se, sp, (flag_rate + sp - 1) / (se + sp - 1)

q = sum(flags) / M
se, sp, pi = estimate(pairs, q)
print(f"\ntrue rate {sum(truth)/M:.3f} | judge flag rate {q:.3f} | validation: sens {se:.2f} spec {sp:.2f}")
print(f"corrected estimate {pi:.3f}")
reps = []
for _ in range(2000):
    bp = [r.choice(pairs) for _ in range(V)]
    bq = sum(r.choices(flags, k=M)) / M
    try: reps.append(min(1, max(0, estimate(bp, bq)[2])))
    except ZeroDivisionError: pass
reps.sort()
print(f"95% bootstrap interval [{reps[int(.025*len(reps))]:.3f}, {reps[int(.975*len(reps))]:.3f}]")
h = sum(t for t, _ in pairs) / V
print(f"humans alone on {V} replies: {h:.3f}")
print("denominator se+sp-1 =", round(se + sp - 1, 3), "(near 0 means the judge carries no signal)")

# ---------- D. repeat the whole procedure to see bias and spread ----------
raw, cor, hum, tru = [], [], [], []
rr = random.Random(99)
for _ in range(1000):
    tr = [rr.random() < 0.12 for _ in range(M)]
    fl = [(rr.random() < SENS) if t else (rr.random() > SPEC) for t in tr]
    vi = rr.sample(range(M), V)
    pp = [(tr[i], fl[i]) for i in vi]
    qq = sum(fl) / M
    tru.append(sum(tr) / M); raw.append(qq); cor.append(estimate(pp, qq)[2]); hum.append(sum(t for t, _ in pp) / V)
for name, xs in (("true rate", tru), ("raw judge flag rate", raw), ("corrected estimate", cor), ("humans only (300)", hum)):
    print(f"{name:20s} mean {statistics.fmean(xs):.3f}  sd {statistics.stdev(xs):.3f}")

For the single dataset, the true rate is 0.119 and the raw flag rate is 0.147, because the judge flags more supported replies than it should. The validation sample estimates sensitivity 0.83 and specificity 0.92, and the corrected estimate is 0.090 with a bootstrap interval of 0.045 to 0.135, which contains the true value. The interval is wide because the correction inherits the uncertainty of two error rates estimated from a few hundred labels, one of which comes from only about 35 truly unsupported replies. The repetition shows the pattern across runs. The raw flag rate averages 0.157 against a true 0.120, a bias that more data would not remove. The corrected estimate averages 0.121, so the correction removes the bias, but its standard deviation of 0.022 is larger than the 0.019 of an estimate from the 300 human labels alone. For a single measurement of one rate, a model judge corrected with a 300-label validation set is not better than just using the 300 human labels. The judge pays off when it scores far more outputs than people can label, when it scores many releases and slices that would each need their own human labels, or when the quantity of interest is a difference between releases for which a stable bias cancels. Each of those uses depends on the error rates staying what they were at validation. Any change to the judge model, its prompt or the kind of outputs changes them, so the validation is rerun on an anchor set: a fixed set of human-labeled outputs that is rescored whenever the judge changes, with the stored outputs of the last two releases rescored under old and new judges so that a change of grader is not mistaken for a change in the application.

A model’s own stated confidence is a separate quantity. If a judge or the application reports a probability, whether that probability matches observed frequencies is a question about probability calibration, treated in its own article; verbal or token-level confidence of a language model is not calibrated by default and has to be checked in the same way as sensitivity and specificity.

Comparing releases

The decision the evaluation exists to support is usually a comparison: is the candidate at least as good as the current release, and where is it worse? Three features of this setting shape the procedure.

The comparison is paired. Both releases run on the same cases, so the quantity of interest is the difference on each case, and the variation that comes from some cases being harder than others cancels from the difference. The comparison is stochastic. Each release is run k times per case, a case’s pass rate is the fraction of runs that pass, and the case-level status can be defined as passing when at least a stated fraction of the runs pass. And the comparison is stratified. The headline difference is a traffic-weighted sum of per-stratum differences, and critical strata are examined on their own, because a weighted average lets a large improvement in a common stratum hide a collapse in a rare one.

Two simple statistics describe the paired result. The first is a paired bootstrap interval for the weighted difference: resample cases with replacement within each stratum, recompute the weighted difference each time, and take the 2.5th and 97.5th percentiles. The glossary entry on the bootstrap explains the resampling idea, and the article on comparing models covers paired comparison and test-set size in depth. The second is the count of fixed and broken cases, meaning cases that fail under the old release and pass under the new one and the reverse. Cases that behave the same under both releases carry no information about the difference, so under the hypothesis that the releases are equally good, each case that changed status is equally likely to have been fixed or broken. With \(m\) changed cases, of which \(b\) were broken, the exact two-sided sign test gives

\[ p = \min\left(1,\; 2\sum_{j=0}^{\min(b,\,m-b)} \binom{m}{j} 2^{-m}\right). \]

This is the exact form of McNemar’s test, and it treats every case equally, so it speaks about the test set and not about traffic when strata have been oversampled.

The program below simulates two releases on 120 cases in the six strata. Per-case pass probabilities are drawn so that cases have different difficulty that both releases share; release B is configured to do slightly better on the common strata, worse on conflict (by 10 points) and much worse on injection (by 20 points). Each case is run five times per release.

import math, random, statistics
from math import comb

# (stratum, traffic weight, cases in test set, mean pass prob of A, change for B)
SPEC = [("standard", .62, 40, .96, +.02), ("late", .12, 20, .94, +.02), ("final_sale", .08, 15, .90, +.02),
        ("missing_info", .10, 15, .85, +.03), ("conflict", .05, 15, .80, -.10), ("injection", .03, 15, .75, -.20)]
K = 5                                       # runs per case and release
rng = random.Random(2)

cases = []                                  # (stratum, per-run pass probability under A and B)
for name, w, nc, pa, delta in SPEC:
    for _ in range(nc):
        a = min(.999, max(.02, rng.betavariate(pa * 6, (1 - pa) * 6)))   # case difficulty, shared by A and B
        b = min(.999, max(.02, a + delta + rng.gauss(0, .03)))
        cases.append((name, a, b))
W = {name: w for name, w, *_ in SPEC}
strata = [s for s, *_ in SPEC]

def runs(p):
    return sum(rng.random() < p for _ in range(K)) / K
rows = [(s, runs(a), runs(b)) for s, a, b in cases]          # per-case pass rate over K runs

def weighted_diff(rs):
    out = 0.0
    for s in strata:
        xs = [(b - a) for st, a, b in rs if st == s]
        out += W[s] * sum(xs) / len(xs)
    return out

def weighted_rate(rs, col):
    return sum(W[s] * statistics.fmean([r[col] for r in rs if r[0] == s]) for s in strata)

print(f"weighted pass rate A {weighted_rate(rows,1):.3f}  B {weighted_rate(rows,2):.3f}  diff {weighted_diff(rows):+.3f}")

def boot(rs, paired, reps=3000):
    r = random.Random(8); ds = []
    for _ in range(reps):
        d = 0.0
        for s in strata:
            g = [x for x in rs if x[0] == s]
            if paired:
                pick = r.choices(g, k=len(g))
                d += W[s] * statistics.fmean([b - a for _, a, b in pick])
            else:
                pa = r.choices(g, k=len(g)); pb = r.choices(g, k=len(g))
                d += W[s] * (statistics.fmean([b for _, _, b in pb]) - statistics.fmean([a for _, a, _ in pa]))
        ds.append(d)
    ds.sort()
    return ds[int(.025 * reps)], ds[int(.975 * reps)]
for name, p in (("unpaired bootstrap", False), ("paired bootstrap", True)):
    lo, hi = boot(rows, p)
    print(f"{name:19s} 95% interval for diff [{lo:+.3f}, {hi:+.3f}]  width {hi-lo:.3f}")

# case-level status: a case 'passes' when at least 4 of 5 runs pass
ok = lambda x: x >= 0.8
fixed = [(s, a, b) for s, a, b in rows if (not ok(a)) and ok(b)]
broken = [(s, a, b) for s, a, b in rows if ok(a) and not ok(b)]
n_disc = len(fixed) + len(broken)
pval = min(1, 2 * sum(comb(n_disc, j) for j in range(0, min(len(fixed), len(broken)) + 1)) / 2 ** n_disc)
print(f"\ncases fixed {len(fixed)}, broken {len(broken)}; exact sign test p = {pval:.3f}")
print(f"\n{'stratum':13s}{'cases':>6s}{'A':>7s}{'B':>7s}{'fixed':>7s}{'broken':>7s}")
for s in strata:
    g = [x for x in rows if x[0] == s]
    print(f"{s:13s}{len(g):6d}{statistics.fmean([x[1] for x in g]):7.3f}{statistics.fmean([x[2] for x in g]):7.3f}"
          f"{sum(x in fixed for x in g):7d}{sum(x in broken for x in g):7d}")

crit = ("conflict", "injection")
lo, hi = boot(rows, True)
gate_overall = lo >= -0.01
gate_crit = all(sum(x in broken for x in rows if x[0] == s) <= sum(x in fixed for x in rows if x[0] == s) for s in crit)
print(f"\ngate: overall lower bound >= -0.010: {gate_overall}; no critical stratum with net breakage: {gate_crit}")
print("decision:", "ship" if gate_overall and gate_crit else "hold: review broken cases")

weighted pass rate A 0.940  B 0.927  diff -0.013
unpaired bootstrap  95% interval for diff [-0.045, +0.020]  width 0.065
paired bootstrap    95% interval for diff [-0.042, +0.015]  width 0.057

cases fixed 4, broken 17; exact sign test p = 0.007

stratum       cases      A      B  fixed broken
standard         40  0.950  0.960      1      3
late             20  0.980  0.970      0      0
final_sale       15  0.947  0.960      1      1
missing_info     15  0.893  0.893      1      1
conflict         15  0.880  0.680      1      5
injection        15  0.827  0.520      0      7

gate: overall lower bound >= -0.010: False; no critical stratum with net breakage: False
decision: hold: review broken cases

The weighted pass rate is 0.940 for A and 0.927 for B, a difference of -0.013. The paired bootstrap interval, -0.042 to +0.015, includes zero, and from the aggregate alone the candidate looks about equal or slightly worse. The sign test on the changed cases tells a different story: 4 cases fixed and 17 broken, p = 0.007. The table says where. The common strata are flat: standard has one fixed and three broken, late none. Conflict and injection carry the damage, with 5 and 7 broken cases and one and none fixed, and the pass rate on injection falls from 0.827 to 0.520. The traffic weights make that collapse nearly invisible in the headline, because injection is 3% of traffic, but the strata where an obeyed instruction approves an invalid return are exactly the ones for which the acceptance criteria were written. The gate in the last lines has two conditions, an overall lower bound no worse than -0.010 and no critical stratum with more broken cases than fixed ones, and it fails both and holds the release for inspection of the broken cases.

Pairing helped, but modestly here: the paired interval has a width of 0.057 and the unpaired one 0.065. In this simulation most of the variation comes from the runs of each case, not from the difference in difficulty between cases, and pairing removes only the latter. The gain is larger when cases differ strongly in difficulty and when more runs per case reduce run noise. The size of the test set limits what any comparison can detect. Card and colleagues examine statistical power in NLP experiments and find that underpowered comparisons are common, which is the same arithmetic as the interval widths above: with 15 cases in a stratum, only a large change is distinguishable from noise, and a gate that requires an interval to exclude zero in every small stratum will rarely pass or fail on evidence. The workable policy is to set per-stratum rules about broken cases, as above, and to add cases to the strata where decisions are close or costly. Dror and colleagues propose a protocol for choosing a significance test for NLP comparisons.

Keeping the regression slice alive

The regression slice is the part of the test set that changes by policy. Every production incident, every escalated failure that is traced to the model, and every bug found in testing becomes a case with a label and a stratum, added after the fix, so that the failure cannot silently return. Cases that fail intermittently should not be dropped. They are measured with their per-case pass rate over k runs and flagged so that a gate on “all regression cases pass” is replaced by one on “no regression case drops below its recorded pass rate by more than the run-to-run noise”. Cases are retired deliberately, with the reason recorded, for example when the policy they test has changed. Each change to the slice is a new test set version, so that every reported comparison names the version it used.

Where a reproducible test set stops

A test set estimates performance on the population its sampling design describes, and the application meets a population that changes. New products, new policy documents and new kinds of customer requests shift the strata and their weights, so the weights are refreshed from current traffic and strata that appear in production but not in the set are added. Even then the set tells nothing about requests that its design could not produce. Offline results also do not capture how people respond to the system in use, and the article on offline metrics and online outcomes describes the gap that online evaluation and controlled rollout exists to close.

The discipline of the set can erode in quiet ways. Repeated looks at the confirmation slice train the team on it, a judge validated once drifts when its model or the outputs it sees change, and a metric that is optimized stops measuring what it measured. The record that makes these visible is a log of which versions of the test set, scorers and judges produced each reported number, and how many candidates have been compared against the same confirmation cases. Evaluation of components with their own structure follows the same principles with additional measures: evaluating retrieval and answers separately adds recall and groundedness, and evaluating agent actions adds trajectories and tool correctness.

References

• Ribeiro, M. T., Wu, T., Guestrin, C., Singh, S. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Proceedings of ACL 2020; arXiv:2005.04118. https://arxiv.org/abs/2005.04118
• Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track; arXiv:2306.05685. https://arxiv.org/abs/2306.05685
• Biderman, S., et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv:2405.14782. https://arxiv.org/abs/2405.14782
• Gebru, T., et al. (2021). Datasheets for Datasets. Communications of the ACM; arXiv:1803.09010. https://arxiv.org/abs/1803.09010
• Jacovi, A., Caciularu, A., Goldman, O., Goldberg, Y. (2023). Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks. arXiv:2305.10160. https://arxiv.org/abs/2305.10160
• Dror, R., Baumer, G., Shlomov, S., Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. Proceedings of ACL 2018. https://aclanthology.org/P18-1128/
• Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., Jurafsky, D. (2020). With Little Power Comes Great Responsibility. Proceedings of EMNLP 2020. https://aclanthology.org/2020.emnlp-main.745/
• scikit-learn documentation, cohen_kappa_score. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html