Prompt and Model Promotion with Evaluation Gates
Consider a hypothetical support assistant whose prompt is edited on a Tuesday. The author runs the team’s evaluation script, sees the average score move from 81 to 84 percent, and updates the production alias. Two days later the support queue shows a rise in tickets from customers who write in other languages, and nobody can say whether the new prompt is responsible, because the evaluation script printed only an average and the old prompt text lives in someone’s editor history. Rolling back takes a day of searching. Each step in this story was reasonable on its own. What was missing is a release process that decides, from recorded evidence and a rule agreed beforehand, whether a change may reach users, and that can undo the decision quickly.
Prompts and models change the behavior of an AI application as much as code does, and they change far more often. A prompt is text edited as casually as a configuration file, and a model alias can be repointed in one line. A platform that serves many teams therefore needs a release gate: a service that takes a candidate change, compares it with the version currently serving traffic on a fixed evaluation set, applies a policy, records the decision, and only then lets the change be promoted gradually. This article builds the logic of such a gate and runs it on synthetic results. By the end you should be able to define a versioned evaluation set and say why per-example results are stored, write a threshold as a non-inferiority rule with a margin, compute and read a paired bootstrap interval, explain why an average can rise while a slice regresses, use an LLM judge without importing its biases, design approvals and a canary with rollback, and say what an offline gate cannot see.
All numbers in this article come from synthetic evaluation results generated by the listed code. They are not results from any real model or prompt. The code uses Python 3.12 and NumPy 2.2 on one machine, and the listed outputs are from the runs shown.
What this article builds on
Model Gateways: Routing, Quotas, and Provider Failover introduced model aliases that map to weighted targets and an audit record that names the model that served each request. Those two mechanisms are the means of promotion and observation here: promoting a change moves an alias, a canary is a weight, and rollback is moving it back. Versioning Models, Data, Prompts, and Evaluation Artifacts explains why prompts, evaluation sets, and models are versioned artifacts with immutable identifiers. This article assumes both, and assumes basic probability: a mean, a standard deviation, and the idea of a confidence interval, which is introduced again below where it is used.
The methodology of building a good evaluation set and of designing online experiments is a subject of its own. This article takes the platform’s side: what a release gate service must store, compute, and enforce, whatever the application is.
The gate as a service
A release gate sits between “a candidate exists” and “traffic reaches it”. It is a sequence of steps, each of which leaves a record.
candidate registered (prompt version / model target / config, all immutable ids)
-> evaluation run on a pinned dataset version, per-example results stored
-> gate policy: floor, non-inferiority vs baseline, slice checks, must-pass cases
PASS -> eligible for canary
INCONCLUSIVE -> named approver decides, reason recorded
BLOCK -> not promotable
-> canary: small traffic share via the gateway, compare with baseline online
-> stage up (5% -> 25% -> 50% -> 100%) or roll back
-> release record: what, against what, which rule, who approved, what happenedPromotion itself needs a registry that separates immutable versions from the mutable names that point at them. MLflow’s Model Registry documents this split: numbered model versions, aliases such as @champion that can be reassigned, and version tags such as pre_deploy_checks: "PASSED" that can carry the gate’s verdict. The same split applies to prompts and gateway aliases.
The division of labor follows the platform boundary used earlier in this course. The application team owns what “good” means: the evaluation set, the metric, and the margins. The platform owns the mechanism: it refuses to compare results from different dataset versions, computes intervals the same way for everyone, keeps the record, and makes promotion and rollback one controlled operation. A platform cannot decide whether a summarizer is good, but it can verify that the thing that was evaluated is the thing that is about to be promoted, that the rule was applied as written, and that the previous version is still there to return to.
Versioned evaluation sets and per-example results
An evaluation set used for gating has to behave like a release artifact. Its examples do not change after it is published; extending it creates a new version. The gate in this article identifies a dataset version by a hash of its manifest, which lists each example’s identifier, slice label, and must-pass flag, so any change to the contents or labels yields a different version string. A comparison between a baseline and a candidate is valid only if both were evaluated on the same version, and the gate refuses to compare otherwise.
Three kinds of structure in the set matter for gating.
- Slices are subsets that correspond to distinct user populations or task types, such as billing questions and requests in other languages. Aggregate scores weight slices by how many examples each contains, which is a property of how the set was assembled and not of how important each group is.
- Must-pass cases are examples where a failure is unacceptable regardless of averages, such as a required refusal or a policy-critical answer. They are checked with a zero-tolerance rule.
- Each example needs a stable identifier so results from different runs can be paired.
The platform should store per-example outcomes, not only the aggregate. The aggregate cannot be used for a paired comparison, cannot be sliced afterward, and cannot be audited when someone asks which cases got worse. Storing one row per example per run (example id, pass or score, run id) is cheap compared with the cost of producing it.
The set also wears out. If a team tunes a prompt repeatedly against one set, the prompt starts to fit that set’s quirks, and its score there overstates its quality elsewhere. A held-out portion that tuning never touches, and periodic refresh from real traffic, keep the gate honest. A refresh is a new dataset version, which means the baseline is re-evaluated on it before comparisons resume.
Thresholds: floor, margin, and three outcomes
Two kinds of threshold appear in practice, and they answer different questions.
An absolute floor asks whether the candidate is good enough on its own, for instance at least 75 percent on the set. A floor is easy to state and ignores what is currently serving. A candidate can clear a floor and still be worse than what it replaces.
A relative rule compares the candidate with the baseline. The naive form, “candidate score at least baseline score”, fails for a reason that comes up in the next sections: scores on a finite set are noisy, so a candidate that is truly worse will pass by chance some fraction of the time. The better form is non-inferiority with a margin. The product owner states the largest loss they would accept, say 3 points. The gate then asks whether the data show that the loss is no larger than the margin.
A confidence interval makes that question precise. An interval for the difference in pass rate between candidate and baseline, written \([\text{lo}, \text{hi}]\), is a range of true differences consistent with the data at the chosen confidence level (95 percent here). With a margin \(m\), the gate sorts results into three outcomes.
- PASS when \(\text{lo} \ge -m\): even the pessimistic end of the interval is within the tolerated loss.
- BLOCK when \(\text{hi} < -m\): even the optimistic end is a loss larger than the margin.
- INCONCLUSIVE otherwise: the interval straddles \(-m\), so the data cannot say whether the loss is acceptable.
The third outcome is the important one. A gate with only pass and fail has to decide something when the evidence is inconclusive, and either choice is wrong a meaningful fraction of the time. INCONCLUSIVE makes the uncertainty a visible state that a person resolves, or that more evaluation data resolves.
Computing the interval: the paired bootstrap
The baseline and the candidate are evaluated on the same examples, so each example yields a pair, and the quantity of interest is the per-example difference \(d_i = c_i – b_i\), where \(c_i\) and \(b_i\) are the candidate’s and baseline’s outcomes (1 for pass, 0 for fail). The mean difference is \(\bar d\). Hard examples are hard for both systems, so most of the variation between examples cancels in the difference, and the paired interval is narrower than one computed from two independent samples. Evan Miller’s paper on error bars for evaluations makes the same point and gives the normal-approximation formulas: the standard error of the mean difference is \(s_d / \sqrt{n}\) for \(n\) examples, where \(s_d\) is the standard deviation of the \(d_i\), and an approximate 95 percent interval is
\[ \bar d \pm 1.96 \, \frac{s_d}{\sqrt{n}} \]
The gate below uses the percentile bootstrap instead, which needs no formula for \(s_d\) and extends to scores that are not binary. The procedure is: draw \(n\) examples from the set with replacement, compute the mean difference of that resample, repeat 10,000 times, and take the 2.5th and 97.5th percentiles of the resampled means as the interval. It treats the observed examples as a stand-in for the population of requests the application will see, so the uncertainty it reports comes from which examples were in the set. Two limits are worth knowing. Examples that are not independent, such as several turns from one conversation or near-duplicate questions, make the interval too narrow unless the resampling unit is the conversation or the cluster. And repeated generations with a nonzero sampling temperature add a second source of noise: running each example several times and averaging reduces generation noise but not the example-sampling noise that the bootstrap reflects.
The gate in code
The following code builds a synthetic versioned evaluation set with four slices (billing, shipping, refunds, and multilingual with only 40 examples) and 30 must-pass cases. It simulates a baseline and three candidates with a shared per-example difficulty, so their outcomes are correlated as real paired results are. It then applies the policy: a 75 percent floor on the candidate’s own score (judged by the lower interval bound), a 3-point margin overall, an 8-point margin per slice, a zero-tolerance rule for must-pass cases, and an automatic INCONCLUSIVE for slices smaller than 30 examples. The policy margins are example values; the product owner’s tolerated loss sets the real ones.
import hashlib, json
from statistics import NormalDist
import numpy as np
# ---- a versioned evaluation set: ids, slice labels, a shared latent difficulty -----------------
SLICES = [("billing", 240), ("shipping", 200), ("refunds", 120), ("multilingual", 40)]
def make_dataset(slices=SLICES, seed=11):
ids, sl = [], []
for name, n in slices:
ids += [f"{name[:3]}-{i:03d}" for i in range(n)]; sl += [name] * n
# must-pass cases: every 12th billing or refunds example (30 of them in both versions of the set)
must_pass = np.array([s in ("billing", "refunds") and int(i[-3:]) % 12 == 0 for i, s in zip(ids, sl)])
manifest = "\n".join(f"{i}:{s}:{int(m)}" for i, s, m in zip(ids, sl, must_pass))
z = np.random.default_rng(seed).standard_normal(sum(n for _, n in slices))
return dict(ids=ids, slices=slices, slice=np.array(sl), must_pass=must_pass, z=z,
version="evalset-" + hashlib.sha256(manifest.encode()).hexdigest()[:12])
def run_system(ds, pass_prob, seed, shared=0.5, force_pass=False, fail_one_must_pass=False):
"""Synthetic outcomes of one system on the set. `shared` is the share of difficulty common to all systems."""
rng = np.random.default_rng(seed)
p = np.array([pass_prob[s] for s in ds["slice"]])
thresh = np.array([NormalDist().inv_cdf(x) for x in p])
latent = np.sqrt(shared) * ds["z"] + np.sqrt(1 - shared) * rng.standard_normal(len(p))
ok = latent < thresh
ok[ds["must_pass"]] = True # in this example the must-pass cases are easy
if fail_one_must_pass: ok[np.flatnonzero(ds["must_pass"])[0]] = False
return dict(ok=ok.astype(float), dataset=ds["version"])
# ---- statistics -----------------------------------------------------------------------------------
def paired_bootstrap(diff, n_boot=10_000, seed=0):
"""95% percentile bootstrap interval for the mean of per-example differences."""
rng = np.random.default_rng(seed)
means = diff[rng.integers(0, len(diff), size=(n_boot, len(diff)))].mean(axis=1)
lo, hi = np.quantile(means, [0.025, 0.975])
return diff.mean(), lo, hi
POLICY = dict(version="policy-3", floor=0.75, overall_margin=0.03, slice_margin=0.08, min_slice_n=30)
def classify(lo, hi, margin):
"""Non-inferiority with a margin: PASS if the loss is credibly within the margin,
BLOCK if it is credibly beyond it, INCONCLUSIVE if the data cannot tell."""
if lo >= -margin: return "PASS"
if hi < -margin: return "BLOCK"
return "INCONCLUSIVE"
def gate(ds, base, cand, policy=POLICY):
if base["dataset"] != cand["dataset"] or base["dataset"] != ds["version"]:
return dict(decision="BLOCK", checks=[("dataset version", "mismatch", "BLOCK")])
checks, diff = [], cand["ok"] - base["ok"]
m, lo, hi = paired_bootstrap(cand["ok"], seed=1)
checks.append(("candidate overall >= floor", f"{m:.3f} [{lo:.3f}, {hi:.3f}]", "PASS" if lo >= policy["floor"] else "BLOCK"))
m, lo, hi = paired_bootstrap(diff, seed=2)
checks.append(("overall vs baseline", f"{m:+.3f} [{lo:+.3f}, {hi:+.3f}]", classify(lo, hi, policy["overall_margin"])))
broke = int(((base["ok"] == 1) & (cand["ok"] == 0) & ds["must_pass"]).sum())
checks.append(("must-pass cases failed", str(broke), "BLOCK" if broke else "PASS"))
for name, _ in ds["slices"]:
mask = ds["slice"] == name
m, lo, hi = paired_bootstrap(diff[mask], seed=3)
status = classify(lo, hi, policy["slice_margin"]) if mask.sum() >= policy["min_slice_n"] else "INCONCLUSIVE"
checks.append((f"slice {name} (n={mask.sum()})", f"{m:+.3f} [{lo:+.3f}, {hi:+.3f}]", status))
states = for c in checks]
decision = "BLOCK" if "BLOCK" in states else "INCONCLUSIVE" if "INCONCLUSIVE" in states else "PASS"
return dict(decision=decision, checks=checks)
def release_record(candidate, baseline, ds, result, policy=POLICY):
rec = dict(candidate=candidate, baseline=baseline, dataset=ds["version"], policy=policy["version"],
decision=result["decision"], checks=[dict(zip(("check", "estimate", "status"), c)) for c in result["checks"]])
rec["record_sha256"] = hashlib.sha256(json.dumps(rec, sort_keys=True).encode()).hexdigest()[:16]
return recThe runner defines the true pass rates for each system, evaluates the three candidates on the 600-example set, then on a second dataset version where the multilingual slice grows from 40 to 200 examples, and prints a release record for the first candidate.
import json
from gate import *
TRUE = { # pass probabilities used to generate the synthetic outcomes
"baseline": dict(billing=.84, shipping=.82, refunds=.78, multilingual=.70),
"A: higher average, multilingual worse": dict(billing=.88, shipping=.87, refunds=.82, multilingual=.55),
"B: clear gain in every slice": dict(billing=.89, shipping=.87, refunds=.83, multilingual=.76),
"C: higher average, fails a must-pass": dict(billing=.88, shipping=.87, refunds=.82, multilingual=.72),
}
def evaluate(ds, verbose):
base = run_system(ds, TRUE["baseline"], seed=100)
print(f"evaluation set {ds['version']}: {len(ds['ids'])} examples, baseline pass rate {base['ok'].mean():.3f}\n")
out = {}
for i, (name, probs) in enumerate(list(TRUE.items())[1:]):
cand = run_system(ds, probs, seed=200 + i, fail_one_must_pass=name.startswith("C"))
out[name] = gate(ds, base, cand)
print(f"candidate {name} -> {out[name]['decision']}")
if verbose:
for check, est, status in out[name]["checks"]:
print(f" {check:<30s} {est:<28s} {status}")
print()
return out
v1 = make_dataset()
res = evaluate(v1, verbose=True)
print("--- the multilingual slice grows from 40 to 200 examples (a new dataset version) ---\n")
v2 = make_dataset([("billing", 240), ("shipping", 200), ("refunds", 120), ("multilingual", 200)])
res2 = evaluate(v2, verbose=True)
rec = release_record("prompt-v8", "prompt-v7", v1, res["A: higher average, multilingual worse"])
print(json.dumps({k: rec[k] for k in ("candidate", "baseline", "dataset", "policy", "decision", "record_sha256")}))evaluation set evalset-117f98e5c35f: 600 examples, baseline pass rate 0.812
candidate A: higher average, multilingual worse -> INCONCLUSIVE
candidate overall >= floor 0.847 [0.817, 0.875] PASS
overall vs baseline +0.035 [-0.002, +0.072] PASS
must-pass cases failed 0 PASS
slice billing (n=240) +0.042 [-0.008, +0.092] PASS
slice shipping (n=200) +0.065 [+0.005, +0.125] PASS
slice refunds (n=120) +0.058 [-0.033, +0.150] PASS
slice multilingual (n=40) -0.225 [-0.400, -0.050] INCONCLUSIVE
candidate B: clear gain in every slice -> INCONCLUSIVE
candidate overall >= floor 0.870 [0.843, 0.895] PASS
overall vs baseline +0.058 [+0.022, +0.095] PASS
must-pass cases failed 0 PASS
slice billing (n=240) +0.054 [+0.004, +0.104] PASS
slice shipping (n=200) +0.065 [+0.000, +0.130] PASS
slice refunds (n=120) +0.092 [+0.000, +0.183] PASS
slice multilingual (n=40) -0.050 [-0.225, +0.125] INCONCLUSIVE
candidate C: higher average, fails a must-pass -> BLOCK
candidate overall >= floor 0.877 [0.850, 0.902] PASS
overall vs baseline +0.065 [+0.030, +0.102] PASS
must-pass cases failed 1 BLOCK
slice billing (n=240) +0.021 [-0.033, +0.075] PASS
slice shipping (n=200) +0.090 [+0.030, +0.150] PASS
slice refunds (n=120) +0.092 [+0.017, +0.167] PASS
slice multilingual (n=40) +0.125 [-0.025, +0.275] PASS
--- the multilingual slice grows from 40 to 200 examples (a new dataset version) ---
evaluation set evalset-a5f6e5a3739f: 760 examples, baseline pass rate 0.783
candidate A: higher average, multilingual worse -> INCONCLUSIVE
candidate overall >= floor 0.786 [0.755, 0.814] PASS
overall vs baseline +0.003 [-0.032, +0.037] INCONCLUSIVE
must-pass cases failed 0 PASS
slice billing (n=240) +0.042 [-0.008, +0.092] PASS
slice shipping (n=200) +0.065 [+0.005, +0.125] PASS
slice refunds (n=120) +0.058 [-0.033, +0.150] PASS
slice multilingual (n=200) -0.140 [-0.220, -0.060] INCONCLUSIVE
candidate B: clear gain in every slice -> PASS
candidate overall >= floor 0.847 [0.822, 0.872] PASS
overall vs baseline +0.064 [+0.030, +0.099] PASS
must-pass cases failed 0 PASS
slice billing (n=240) +0.054 [+0.004, +0.104] PASS
slice shipping (n=200) +0.065 [+0.000, +0.130] PASS
slice refunds (n=120) +0.092 [+0.000, +0.183] PASS
slice multilingual (n=200) +0.060 [-0.010, +0.130] PASS
candidate C: higher average, fails a must-pass -> BLOCK
candidate overall >= floor 0.841 [0.814, 0.866] PASS
overall vs baseline +0.058 [+0.025, +0.091] PASS
must-pass cases failed 1 BLOCK
slice billing (n=240) +0.021 [-0.033, +0.075] PASS
slice shipping (n=200) +0.090 [+0.030, +0.150] PASS
slice refunds (n=120) +0.092 [+0.017, +0.167] PASS
slice multilingual (n=200) +0.050 [-0.020, +0.120] PASS
{"candidate": "prompt-v8", "baseline": "prompt-v7", "dataset": "evalset-117f98e5c35f", "policy": "policy-3", "decision": "INCONCLUSIVE", "record_sha256": "eb210baf1de5254c"}Reading the gate’s decisions
Candidate A is the case the opening story describes. On the first dataset version its overall pass rate rises by 3.5 points, with an interval of \([-0.2, +7.2]\) points, which satisfies the 3-point non-inferiority rule. A reviewer looking only at the average would approve it. The slice view shows that the multilingual slice dropped by 22.5 points. That slice has 40 examples and the interval is wide, \([-40.0, -5.0]\) points, and since its upper end is above the slice margin of \(-8\) points, the gate returns INCONCLUSIVE instead of BLOCK. It cannot say the loss is beyond the margin, and it cannot say it is within it, and it refuses to treat the overall improvement as a reason to promote. A person has to decide, with the numbers in front of them.
Candidate B improves in every slice by design, and still gets INCONCLUSIVE on the first set. The reason is the same 40-example slice. The 40 examples happened to show a 5-point drop for a candidate that is better there, with an interval of \([-22.5, +12.5]\) points, wide enough that nothing can be concluded whatever the true change. A slice that small can never support an automatic decision under this policy. The gate is telling the application team that its evaluation set cannot verify this population.
Candidate C has the best average and fails one must-pass case that the baseline passed. The must-pass rule blocks it outright. Its other numbers do not matter, which is the purpose of a zero-tolerance rule: a high average cannot buy back a failure the product has said it will not accept.
The second dataset version changes the outcomes. With 200 multilingual examples, candidate B passes every check. Candidate A’s overall gain shrinks to +0.3 points, because the weak slice now has more weight, and its overall interval \([-3.2, +3.7]\) straddles the margin, so the overall check also moves to INCONCLUSIVE. The multilingual interval narrows to \([-22.0, -6.0]\) points, and its upper end is still above the 8-point margin, so the gate continues to ask for a decision. The evidence is now much clearer for the reviewer: the loss is somewhere between 6 and 22 points. The results show that growing the weak slice, not tuning the threshold, was what made the gate useful. The dataset version string changed, as it should.
How many examples a gate needs
The decisions above depend on the number of examples. The next simulation makes that dependence explicit. It draws 4,000 paired evaluations for each combination of dataset size and true change in pass rate (the baseline passes 80 percent of examples), and records how often the naive rule “candidate point estimate at least baseline” and the interval rule (3-point margin, 95 percent normal-approximation interval) reach each decision (INCONCL. means inconclusive).
import numpy as np
from math import sqrt
from statistics import NormalDist
rng = np.random.default_rng(5)
MARGIN, BASE, SHARED, REPS = 0.03, 0.80, 0.5, 4000
def outcomes(n, true_diff):
"""REPS paired evaluations of n examples; the candidate's true pass rate is BASE + true_diff."""
z, eb, ec = (rng.standard_normal((REPS, n)) for _ in range(3))
xb, xc = sqrt(SHARED) * z + sqrt(1 - SHARED) * eb, sqrt(SHARED) * z + sqrt(1 - SHARED) * ec
ok_b = xb < NormalDist().inv_cdf(BASE)
ok_c = xc < NormalDist().inv_cdf(BASE + true_diff)
d = ok_c.astype(float) - ok_b
m, se = d.mean(axis=1), d.std(axis=1, ddof=1) / sqrt(n)
return (m >= 0).mean(), ((m - 1.96 * se) >= -MARGIN).mean(), ((m + 1.96 * se) < -MARGIN).mean()
print("share of 4,000 simulated evaluations that end in each decision (baseline pass rate 0.80, margin 3 points)\n")
print(f"{'true change':>11s} {'examples':>9s} {'point estimate >= 0':>20s} {'gate PASS':>10s} {'gate BLOCK':>11s} {'gate INCONCL.':>12s}")
for d in (-0.06, -0.03, 0.0, 0.03):
for n in (100, 300, 1000, 3000):
naive, ok, bad = outcomes(n, d)
print(f"{d:+11.2f} {n:9d} {naive:20.0%} {ok:10.0%} {bad:11.0%} {1 - ok - bad:12.0%}")
print()share of 4,000 simulated evaluations that end in each decision (baseline pass rate 0.80, margin 3 points)
true change examples point estimate >= 0 gate PASS gate BLOCK gate INCONCL.
-0.06 100 13% 1% 9% 90%
-0.06 300 2% 0% 18% 82%
-0.06 1000 0% 0% 48% 52%
-0.06 3000 0% 0% 90% 10%
-0.03 100 30% 2% 3% 95%
-0.03 300 14% 2% 3% 95%
-0.03 1000 3% 3% 2% 95%
-0.03 3000 0% 2% 3% 95%
+0.00 100 55% 9% 1% 90%
+0.00 300 52% 20% 0% 80%
+0.00 1000 52% 52% 0% 48%
+0.00 3000 51% 93% 0% 7%
+0.03 100 77% 25% 0% 75%
+0.03 300 88% 61% 0% 39%
+0.03 1000 98% 98% 0% 2%
+0.03 3000 100% 100% 0% 0%The naive rule accepts truly worse candidates a surprising fraction of the time when the set is small. With 100 examples, a candidate that is truly 3 points worse passes 30 percent of the time, and one that is 6 points worse passes 13 percent. The interval rule passes those candidates 2 percent and 1 percent of the time. The price shows in the middle rows: with 100 examples, a candidate that is exactly as good as the baseline passes the interval rule only 9 percent of the time and lands in INCONCLUSIVE 90 percent of the time. The set is too small to prove non-inferiority, so everything needs a person. With 3,000 examples, equal candidates pass 93 percent of the time and a candidate that is truly 6 points worse is blocked 90 percent of the time.
A rough rule explains the numbers. To show non-inferiority for a candidate that is truly equal to the baseline, the interval’s half-width must be smaller than the margin \(m\), so
\[ n \gtrsim \left( \frac{1.96 \, s_d}{m} \right)^2 \]
where \(s_d\) is the standard deviation of the paired differences. For pass-fail outcomes \(s_d\) depends on how often the two systems disagree on an example. If they disagree on about 22 percent of examples, as the simulated systems do, \(s_d \approx 0.47\), and a 3-point margin needs roughly \((1.96 \times 0.47 / 0.03)^2 \approx 940\) examples for the interval to just fit, and about 2,600 for a 90 percent chance of passing. These match the simulation’s 52 percent at 1,000 examples and 93 percent at 3,000. A tight margin, a small set, and a noisy metric together make the gate unusable. The margin should be set from what the product can tolerate, and the evaluation set sized so that the margin is verifiable.
Slices need the same arithmetic separately, with their own sizes. A set of 3,000 examples that is 98 percent one slice leaves the other slices at 20 or 30 examples each, and the gate will return INCONCLUSIVE for them every time. With many slices, a second effect appears. If each of ten slices is judged with a 5 percent error rate, the chance that at least one looks like a regression by chance alone is \(1 – 0.95^{10}\), about 40 percent. The INCONCLUSIVE state absorbs this, because a flagged slice leads to a look at the evidence, not to an automatic block. It is also a reason to treat the slice list as a screening tool and to keep the BLOCK rule for slices strict enough to be meaningful.
Using an LLM as the judge
Many tasks have no exact answer to compare with, and the evaluation uses a language model as a judge that labels each output as acceptable or not, or prefers one of two. Zheng and colleagues, who introduced MT-Bench and studied this practice, found that strong judges agree with human preferences more than 80 percent of the time, about the level at which humans agree with each other, and also documented systematic biases: toward the answer in a particular position, toward longer answers (verbosity bias), and toward answers the judge itself generated (self-enhancement). A judge is a measuring instrument with errors, and the gate needs to know what they are.
Two consequences matter for a platform. First, the judge is part of the evaluation configuration: its model, its prompt, and its decoding settings are versioned artifacts, and changing any of them changes what a score means, so the baseline has to be re-scored under the new judge before a comparison is valid. Second, random judge errors only add noise and widen intervals, but errors that differ between the two systems shift the measured difference. More examples do not remove a shift.
The next simulation sets up the second case. Baseline and candidate are equally good, with true pass rates of about 80 percent. The judge passes 95 percent of good answers. On bad answers it passes 10 percent of the baseline’s and 25 percent of the candidate’s, which mimics a judge that is more lenient toward the candidate’s longer answers. A human-labeled audit of the first 400 examples measures the judge’s rates for each system, and the audit rates are used to correct the observed pass rates by inverting \(\text{observed} = \text{tpr} \cdot p + \text{fpr} \cdot (1 – p)\), where tpr is the judge’s pass rate on good answers, fpr its pass rate on bad ones, and \(p\) the true pass rate.
import numpy as np
from math import sqrt
from statistics import NormalDist
rng = np.random.default_rng(21)
N, AUDIT = 2000, 400
TRUTH_RATE = 0.80 # both systems are equally good in reality
TPR, FPR_BASE, FPR_CAND = 0.95, 0.10, 0.25 # the judge is more lenient toward the candidate's (longer) answers
z, e1, e2 = rng.standard_normal(N), rng.standard_normal(N), rng.standard_normal(N)
t = NormalDist().inv_cdf(TRUTH_RATE)
truth_b, truth_c = (sqrt(.5) * z + sqrt(.5) * e1) < t, (sqrt(.5) * z + sqrt(.5) * e2) < t
def judge(truth, fpr):
return np.where(truth, rng.random(N) < TPR, rng.random(N) < fpr)
jb, jc = judge(truth_b, FPR_BASE), judge(truth_c, FPR_CAND)
def diff_ci(c, b):
d = c.astype(float) - b
m, se = d.mean(), d.std(ddof=1) / sqrt(len(d))
return f"{m:+.3f} [{m - 1.96 * se:+.3f}, {m + 1.96 * se:+.3f}]"
print(f"true pass rates (known only because this is a simulation): baseline {truth_b.mean():.3f}, candidate {truth_c.mean():.3f}")
print(f"LLM judge on all {N} examples: baseline {jb.mean():.3f}, candidate {jc.mean():.3f}, difference {diff_ci(jc, jb)}")
# audit: humans label the first AUDIT examples, which lets us measure the judge itself
a = slice(0, AUDIT)
cal = {}
for name, j, tr in (("baseline", jb, truth_b), ("candidate", jc, truth_c)):
tpr, fpr = j[a][tr[a]].mean(), j[a][~tr[a]].mean()
cal[name] = (tpr, fpr)
print(f"audit of {name}: judge passes {tpr:.2f} of the answers humans passed and {fpr:.2f} of the answers humans failed (n={(~tr[a]).sum()} failures)")
fix = lambda obs, c: (obs - c[1]) / (c[0] - c[1]) # invert obs = tpr * p + fpr * (1 - p)
cb, cc = fix(jb.mean(), cal["baseline"]), fix(jc.mean(), cal["candidate"])
print(f"pass rates corrected with each system's own audit rates: baseline {cb:.3f}, candidate {cc:.3f}, difference {cc - cb:+.3f}")true pass rates (known only because this is a simulation): baseline 0.797, candidate 0.800
LLM judge on all 2000 examples: baseline 0.774, candidate 0.819, difference +0.044 [+0.021, +0.068]
audit of baseline: judge passes 0.93 of the answers humans passed and 0.08 of the answers humans failed (n=77 failures)
audit of candidate: judge passes 0.97 of the answers humans passed and 0.25 of the answers humans failed (n=75 failures)
pass rates corrected with each system's own audit rates: baseline 0.812, candidate 0.789, difference -0.023The judge reports a 4.4-point improvement with an interval of \([+2.1, +6.8]\) that excludes zero, for two systems that are equally good. A gate based on this result would call the candidate better with confidence. The audit exposes the cause: the judge passes 8 percent of the failures it sees from the baseline and 25 percent from the candidate. Correcting with each system’s own audit rates brings the difference to \(-2.3\) points, near the truth of zero. Each audit rate rests on roughly 75 failures here, so the corrected figures carry noise of several points. The correction narrows the problem, and it does not eliminate it, so the sound practice is to keep a human-labeled sample in every evaluation run, to monitor the judge’s agreement with it per system, and to treat a gap between systems as a reason to distrust the judge result until it is explained. For pairwise comparisons, swapping the order of the two answers and requiring a consistent verdict addresses position bias. A judge from the same family as the candidate should not be the only judge for that candidate.
Approvals and waivers
An INCONCLUSIVE result needs a named human decision, and the platform defines how it is made. An approval is a record with the approver’s identity, the decision, the reason, and the evidence they were shown, tied to the specific release record and policy version. A few rules keep it meaningful.
- The author of the change cannot be its only approver. Separation of duties is the point of review.
- The approver should be someone accountable for the affected users. For a slice regression, that is the owner of the slice’s product area, who may know that the slice is rare or that the loss is on a case they care about.
- A waiver of a failed check has an expiry and an owner. “We accepted the multilingual loss until the next release, and committed to fixing it” is a decision. A waiver without an end date becomes the new baseline.
- Some checks are not waivable. A must-pass failure on a safety-critical case blocks, and an override needs a higher authority and shows up in the record.
- An emergency path exists, because a hotfix for a harmful prompt cannot wait for a full evaluation. It skips the wait and not the record: the change is promoted with an expedited flag, and the evaluation runs afterward and is attached.
The release record is immutable and hashed. The code prints the hash of the first candidate’s record as record_sha256, which lets a later reader detect an edited record. The record names the dataset version, policy version, baseline and candidate identifiers, each check’s estimate and status, and the final decision. Together with the gateway’s audit log, it lets an investigator answer “what evidence was there when this went live” months later.
Canary, stages, and rollback
An offline gate decides whether a candidate may start receiving traffic. It does not prove the candidate is good in production, so promotion proceeds in stages. Canary Deployment describes the idea: expose a limited part of real work to the candidate while the current release handles the rest, and define the routing, comparison metrics, observation time, and stop conditions beforehand. The mechanism is the weighted alias of the gateway article: the candidate’s target gets 5 percent of the alias’s traffic, then 25, 50, and 100, and a stage advances only when the comparison with the baseline allows it. Progressive delivery controllers such as Argo Rollouts implement the same pattern for Kubernetes workloads. A Rollout’s canary steps create analysis runs from a template that defines the metrics, frequency, and the values considered successful or failed, and the update progresses or aborts depending on the result.
Two details from Google’s SRE Workbook chapter on canarying apply directly. A canary’s problems are hard to see in overall metrics, because a small canary population dilutes them: a candidate with a 20 percent error rate serving 5 percent of traffic raises the overall error rate by only 1 point. The comparison therefore has to be per version, between the candidate and the baseline on comparable traffic. And the canary’s duration and population have to fit the pace of releases, since a canary that lasts a week does not suit a team that ships daily.
The statistics of a canary are the same as the gate’s, with a different sample. The next simulation uses an online success rate of 80 percent for the baseline and a rule that rolls back when the upper end of the 95 percent (one-sided) interval for the candidate’s difference falls below \(-2\) points, meaning the candidate is credibly worse than the margin. It runs three stages: 5 percent of 4,000 requests, 25 percent of 8,000, and 50 percent of 8,000. For each true gap it runs 5,000 rollouts.
import numpy as np
from math import sqrt
rng = np.random.default_rng(8)
P0, MARGIN, REPS = 0.80, 0.02, 5000 # baseline online success rate, tolerated loss
STAGES = [(0.05, 4000), (0.25, 8000), (0.50, 8000)] # (share of traffic to the candidate, requests in the stage)
def rollout(true_gap):
exposed = 0
for k, (share, total) in enumerate(STAGES, 1):
nc = int(share * total); nb = total - nc
pc, pb = rng.binomial(nc, P0 + true_gap) / nc, rng.binomial(nb, P0) / nb
se = sqrt(pc * (1 - pc) / nc + pb * (1 - pb) / nb)
exposed += nc
if (pc - pb) + 1.645 * se < -MARGIN: # credibly worse than the margin: roll back
return k, exposed
return 0, exposed # 0 = reached 100%
print(f"{'true gap':>9s} {'rolled back':>26s} {'completed':>10s} {'candidate requests served':>27s}")
print(f"{'':>9s} {'stage 1':>8s} {'stage 2':>8s} {'stage 3':>8s} {'':>10s} {'mean':>14s}")
for gap in (-0.08, -0.04, -0.02, 0.0):
res = [rollout(gap) for _ in range(REPS)]
stage = [sum(1 for k, _ in res if k == s) / REPS for s in (1, 2, 3, 0)]
print(f"{gap:+9.2f} {stage[0]:8.0%} {stage[1]:8.0%} {stage[2]:8.0%} {stage[3]:10.0%} {np.mean([e for _, e in res]):14.0f}") true gap rolled back completed candidate requests served
stage 1 stage 2 stage 3 mean
-0.08 58% 42% 0% 0% 1035
-0.04 15% 49% 25% 11% 3359
-0.02 4% 5% 5% 87% 5769
+0.00 1% 0% 0% 99% 6158A candidate that is 8 points worse is rolled back at stage 1 in 58 percent of rollouts and at stage 2 in the rest, and it serves a mean of 1,035 requests before the rollback. A candidate that is 4 points worse is caught at some stage 89 percent of the time, but it completes the rollout 11 percent of the time and exposes about 3,400 requests on average. A candidate that is exactly at the margin (2 points worse) completes 87 percent of the time, which is the intended behavior of a rule that tolerates losses up to the margin. The false rollback rate for an equal candidate is 1 percent. A small first stage limits exposure, but its power to detect a moderate loss is low: 200 candidate requests at stage 1 cannot see a 4-point drop reliably. Each stage trades exposure for detection power, and the stage sizes and thresholds should be chosen from this arithmetic and from how costly a bad answer is.
Online comparison also needs guard metrics that do not depend on a quality judgment: error rate, latency percentiles, refusal rate, output length, and cost per request. The gateway’s audit records provide those per served model. Online quality proxies, such as thumbs-down rates, task completion, or escalation to a human, arrive more slowly and more noisily than offline scores, and some are available only after a delay. The canary’s observation time must be long enough for the slowest metric the rollback rule depends on.
Rollback states the conditions under which returning to the previous version works: the old version must still be retained, compatible with current data, interfaces, and access requirements, and the switch must be tested. For a prompt, compatibility includes the tool contract it was written against. For a platform release, the practical rules are to keep the previous target registered and warm enough to take the traffic, to make the rollback a single alias or weight change with the same authority checks as promotion, and to verify afterward which version is serving requests, since an alias repointed in configuration does not help if some processes keep serving the previous model or a cache holds answers generated by the failed version. Rolling back also does not undo what the failed version already did, which matters more once models trigger actions, the subject of the next article.
What an offline gate cannot see
The gate evaluates a fixed set that the platform and the application team believe represents the application. Production differs in four ways that the gate cannot detect.
- Distribution shift. Users change what they ask, and the set ages. A candidate that is better on last quarter’s questions may not be better on this quarter’s.
- Interaction effects. Multi-turn behavior, tool calls with real latency and failures, and long contexts are hard to capture in single-turn examples.
- Metric gap. An offline score approximates something users care about, and the approximation can drift from it. Judges and graders have their own errors.
- Rare and severe cases. A failure that happens once in ten thousand requests will not appear in a few hundred examples, and a canary at 5 percent for an hour may not see it either. These need targeted must-pass cases, red-team sets, and monitoring.
The remedy is a loop. Production failures, escalations, and user corrections feed new examples into the next dataset version, with slice labels and must-pass flags where they belong. The next gate then evaluates both systems on the larger set. Each dataset version also gives a new baseline score, so scores from different versions are not compared with each other.
The gate has failure modes of its own. A team that tunes a prompt until the gate passes is fitting the set. A flaky evaluation that returns different answers on rerun teaches people to rerun until it passes; the remedy is to fix seeds where possible, average repeated generations, and make the gate’s input a stored result and not a live call. A slow or unreliable gate invites bypasses, which is why the emergency path is explicit and recorded. And thresholds drift when waivers accumulate; reviewing waivers and the distribution of INCONCLUSIVE outcomes shows whether the policy still means what it says.
What comes next
Everything so far releases a model that answers. An agent runtime releases a model that acts: it calls tools with credentials, changes state, and can repeat an action when a request is retried. A bad release then costs more than a bad answer, and the gate, the canary, and the rollback in this article are necessary but not sufficient. Agent Runtime Isolation, Tools, and Execution Policies covers the controls that bound what an agent can do while a release is being evaluated and after it is live.
References
- Evan Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640)
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)
- Google SRE Workbook, Canarying Releases
- Argo Rollouts documentation, Analysis and Progressive Delivery
- MLflow documentation, Model Registry
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.