Online Evaluation and Controlled Rollout

Consider a hypothetical team that maintains the returns assistant from the earlier articles. Its next release has two changes: a prompt that makes answers shorter, and a re-indexed policy corpus that adds the new bundle-return rules. The offline gate passes. The weighted pass rate is 0.93, no critical stratum has more broken cases than fixed ones, and the behavioral suite is green. Someone proposes switching all traffic over on Monday.

Three questions remain after the gate. Does the offline number predict what customers experience, given that customers rephrase, retry, and write requests nobody put in the test set? Did the customers who got a decision complete their returns, or did they come back a week later because the decision was wrong? And if the answer to either is bad, how quickly will the team know, how many customers will have been affected by then, and what exactly does “go back” mean for a system that is a prompt, a model, an index, and a set of tools at the same time?

Online evaluation is the measurement of a release on live traffic, and a controlled rollout is the way of exposing live traffic to it in stages so that the measurement can stop the release before the damage is large. It builds on the acceptance criteria and error costs from the framing article and on the offline instrument from the article on reproducible test sets. Everything measured online is also a copy of customer text or behavior, so the article on privacy and output controls is a prerequisite: its split between a metadata tier kept by default and a content tier kept by exception, and its inventory of stores with lifetimes, decide what the evaluation below is allowed to keep. The programs are Python 3.12 using only the standard library. All data is synthetic, written for this article, and every user and model behavior in them is a configured setting, so the numbers show how the methods behave and say nothing about any real system.

Where offline and online results part

An offline test set is a fixed sample of requests with expected behavior, run through a release and scored by code or judges. An online evaluation looks at a different object: the stream of requests that customers actually send, answered once each, by a system that customers react to, with outcomes that arrive days later. The differences are not a single “gap”. They have separate causes, and each cause has its own measurement and its own remedy.

Cause	What changes between offline and online	Returns-assistant example	Response
Population shift	Traffic mix and wording drift from the sampling design	A bundle-return feature creates a request type with no test cases	Refresh stratum weights from traffic; add strata; monitor the input mix
Unseen inputs	Requests the design could not produce	Pasted order emails, mixed languages, photographs described in text	Sample production for the representative slice; keep a stress slice
User reaction	People respond to the output, and the response changes what happens next	A terse refusal makes customers rephrase three times or ask for a person	Measure the session or task, not the single reply
State and interaction	Context, memory, caches, and concurrent traffic	A cached answer for one policy version is served after the policy changed	Include state and caches in the release; test them online
One draw per request	Offline runs give k samples per case; a live request gets one	The same question gets a different decision on a second try	Plan for unpaired comparisons; store outputs
Delayed outcomes	The result is known later	A refund is reversed five days after the chat ended	Define maturity; compare cohorts at equal age
Environment	Provider load, timeouts, fallbacks, rate limits	A slow period routes 3% of requests to the rule-based fallback	Log which path answered; evaluate the blend
Scorer shift	Judges and checks were validated on offline outputs	A judge that was fine on test outputs misreads a new answer style	Re-audit judges on production samples

The first two rows are statements about weights and coverage, and weights can be recomputed. The sampling design in the reproducible-evaluation article estimates \(\hat p = \sum_s W_s \hat p_s\), the traffic-weighted sum of per-stratum pass rates, where \(W_s\) is the share of traffic in stratum \(s\) and \(\hat p_s\) the observed pass rate on its test cases. The estimate describes the traffic mix that the weights were taken from. When the mix changes, the same per-stratum rates give a different overall rate, and a stratum with no test cases gives no rate at all. The program below applies the offline rates of the candidate release to the traffic mix of a later month, in which the bundle-return feature supplies 11% of requests.

# Offline estimate versus production estimate under traffic-mix shift and an unseen stratum.
STRATA = ["standard", "late", "final_sale", "missing_info", "conflict", "injection", "multi_item"]
# offline test set: per-stratum pass rate of release B measured on the set (multi_item has no cases)
OFFLINE_RATE = {"standard": .96, "late": .94, "final_sale": .90, "missing_info": .85,
                "conflict": .80, "injection": .75}
DESIGN_W = {"standard": .62, "late": .12, "final_sale": .08, "missing_info": .10,
            "conflict": .05, "injection": .03, "multi_item": 0.0}
# traffic observed this month: a new bundle-return feature created the multi_item stratum
PROD_W = {"standard": .52, "late": .11, "final_sale": .08, "missing_info": .10,
          "conflict": .05, "injection": .03, "multi_item": .11}

def weighted(rates, w, unknown_rate):
    return sum(w[s] * rates.get(s, unknown_rate) for s in STRATA)

offline = weighted(OFFLINE_RATE, DESIGN_W, 0.0)
print(f"offline estimate with design weights:            {offline:.3f}")
# keep the offline rates, re-weight to current traffic; the unseen stratum has no data
for u in (0.90, 0.70, 0.50):
    print(f"re-weighted to current traffic, multi_item rate {u:.2f}: {weighted(OFFLINE_RATE, PROD_W, u):.3f}")
known_share = 1 - PROD_W["multi_item"]
only_known = sum(PROD_W[s] * OFFLINE_RATE[s] for s in OFFLINE_RATE) / known_share
print(f"share of traffic with no offline cases: {PROD_W['multi_item']:.2f}; "
      f"known-strata estimate renormalized: {only_known:.3f}")
lo = weighted(OFFLINE_RATE, PROD_W, 0.0); hi = weighted(OFFLINE_RATE, PROD_W, 1.0)
print(f"worst/best case for the unseen stratum: [{lo:.3f}, {hi:.3f}]  (width {hi-lo:.3f})")

The output:

offline estimate with design weights:            0.927
re-weighted to current traffic, multi_item rate 0.90: 0.921
re-weighted to current traffic, multi_item rate 0.70: 0.899
re-weighted to current traffic, multi_item rate 0.50: 0.877
share of traffic with no offline cases: 0.11; known-strata estimate renormalized: 0.924
worst/best case for the unseen stratum: [0.822, 0.932]  (width 0.110)

The offline estimate of 0.927 uses the weights from when the test set was designed. Re-weighting the same per-stratum rates to the current mix, and assuming the new stratum behaves as well as the average stratum (0.90), gives 0.921. If the unseen stratum passes only 0.70 of the time, the overall rate is 0.899, and at 0.50 it is 0.877. Nothing about the release changed between these lines. The test set’s coverage and the traffic changed, and the estimate moved by up to five points. The bounds in the last line say what the offline data alone supports: with 11% of traffic uncovered, the overall rate lies anywhere in an interval 0.110 wide, which is wider than most differences a release gate is asked to detect. The remedy is partly offline, by adding cases for the new stratum, and partly online, because production is the only place where the actual rate for that stratum can be observed without writing the cases first. The article on offline metrics and online outcomes develops the other part of the gap that applies to any model, namely that exposure and action capacity change what a score is worth.

Two of these causes are specific to generative applications: the single draw per request and the shift in scorers.

One draw per request, and no pairing

The offline comparison in the reproducible-evaluation article is paired: both releases run on the same cases, each case several times, and the difference is computed case by case, so the variation from some cases being hard cancels out. A live request is answered once, by one release. A customer assigned to the candidate never sees what the current release would have said, so there is no case-level difference to compute. The comparison between arms is a comparison between two independent samples of customers, and its interval is that of two independent proportions. Run-to-run randomness of the model does not need separate treatment, since each session already contains one draw and the variance of a proportion, \(p(1-p)/n\), counts it together with the variation in case difficulty. What is lost is the pairing, which is why online comparisons need more sessions than offline ones to detect the same difference, a cost quantified in the section on exposure.

A related consequence is that a live answer cannot be regenerated. Output text, the release fingerprint, the retrieved evidence ids, and the tool calls of every sampled request have to be stored when they happen, in the tier that the privacy article allows, or an investigation that starts a week later has nothing to examine. Counterfactual replay, in which the logged inputs of a production sample are sent to a candidate release offline, is still useful as a screen, and a redacted production sample is the best source of cases for the representative slice. It shows what the candidate says to real inputs. It does not show how customers react to what it says.

Scorer shift

Judges and deterministic checks are validated against a set of outputs. When the release changes style, for example by producing shorter answers, the outputs a judge sees online come from a different distribution than the ones it was validated on, and its sensitivity and specificity can change with them. A judge that is applied to production samples is therefore audited the same way as offline: a sample of its verdicts is compared with human labels, per failure type. The correction and validation procedure in the reproducible-evaluation article applies unchanged. The new element online is that the anchor set of human-labeled outputs has to be refreshed with production outputs of the current release, or the judge is validated on text it no longer sees.

Defining the task outcome

Online measurements form a ladder. The lower rungs are cheap, immediate, and far from what the consumer of the output cares about; the upper rungs are close to that goal and slow or expensive.

Level	Examples for the returns assistant	What it tells	Weakness
System health	Latency percentiles, error rate, schema-valid rate, share answered by the fallback	The service works	Says nothing about whether answers are right
Response properties	Abstention rate, citation present, answer length	The output has the intended form	A well-formed answer can be wrong
User behavior	Rephrase within one minute, request for a person, copy of the answer, thumbs-down, abandonment	A customer reacted	Biased and ambiguous (see the next section)
Task outcome	Return completed correctly; no contact about the same order within 7 days; no refund reversal	The task was done	Delayed; needs a system of record or an audit
Business result	Cost per successfully handled return, refund leakage, satisfaction survey	The workflow pays off	Slow, confounded by everything else that changed

The acceptance criteria written during framing are phrased at the task-outcome level, so the online evaluation has to be too. Lower rungs are used for what they are good at. System health and response properties are guardrails that can stop a rollout within minutes, and behavior signals point at cases to read. A claim that a release is better needs an outcome from the fourth rung or an audit that stands in for it.

What an outcome definition must fix

An outcome metric is a ratio with a unit, a numerator, a denominator, a window, and a source of truth, and each of these is a decision. For the returns assistant, one definition is: among return conversations that started at least 7 days ago (the unit is the conversation, the denominator is the matured cohort), the share in which the return was completed and the customer did not contact support about the same order within 7 days (the numerator), with completion and contacts read from the order system and the support system and not from the model’s own claim. Conversations from out-of-scope languages, as defined in the task boundary, are excluded and counted separately, so the exclusion cannot hide a shift in who is served.

The unit matters for the statistics as well. Requests from one customer are not independent, so the unit of analysis is the unit of assignment, usually the customer or the conversation, and the variance inflation described in the article on A/B testing and experiment design applies to request-level counts.

Immature outcomes

An outcome that needs seven quiet days is unknown for every conversation younger than seven days. Averages computed on whatever has arrived are biased in a direction that depends on how the unknowns are counted. The program below builds a synthetic log of 12,000 conversations over 28 days, in which 90% of returns complete and 12% of completed returns lead to a re-contact after a delay of up to seven days. It then evaluates the success rate as of the last day, in four ways.

import random

# Synthetic event log for a returns assistant. One row per return conversation (the unit).
# Outcome: success = the return was completed AND the customer did not contact support
# about the same order within 7 days of the conversation start.
rng = random.Random(4)
DAYS = 28                      # conversations start on days 0..27; "today" is the end of day 27
TODAY = DAYS - 1
rows = []
for _ in range(12000):
    start = rng.randrange(DAYS)
    completed = rng.random() < 0.90
    recontact = completed and rng.random() < 0.12          # 12% of completed returns lead to a re-contact
    delay = min(7, int(rng.expovariate(1 / 2.5))) if recontact else None
    rows.append((start, completed, None if delay is None else start + delay))

def known_outcome(row, asof):
    """Return 'fail', 'success' or None (not yet known) as of day `asof`."""
    start, completed, recontact_day = row
    if not completed:
        return "fail"
    if recontact_day is not None and recontact_day <= asof:
        return "fail"
    return "success" if start + 7 <= asof else None

n = len(rows)
final = sum(known_outcome(r, 10**6) == "success" for r in rows) / n
print(f"rate with every outcome fully observed (only possible in simulation): {final:.3f}")

kn = [known_outcome(r, TODAY) for r in rows]
succ, fail, unk = kn.count("success"), kn.count("fail"), kn.count(None)
print(f"as of today: success {succ}, fail {fail}, not yet known {unk} of {n}")
print(f"A. unknown counted as failure:   {succ / n:.3f}")
print(f"B. unknown counted as success:   {(succ + unk) / n:.3f}")
print(f"C. only conversations with a known outcome: {succ / (succ + fail):.3f}")
mature = [r for r in rows if r[0] + 7 <= TODAY]
m = sum(known_outcome(r, TODAY) == "success" for r in mature)
print(f"D. cohort that started at least 7 days ago: {m}/{len(mature)} = {m / len(mature):.3f}")

The output:

rate with every outcome fully observed (only possible in simulation): 0.792
as of today: success 7165, fail 2405, not yet known 2430 of 12000
A. unknown counted as failure:   0.597
B. unknown counted as success:   0.800
C. only conversations with a known outcome: 0.749
D. cohort that started at least 7 days ago: 7165/9017 = 0.795

The simulation knows the true rate, 0.792, because it can look past the end of the window. Method A counts conversations with an unknown outcome as failures and reports 0.597, which would trigger a rollback of a perfectly healthy release. Method B counts them as successes and reports 0.800, which hides every re-contact that has not happened yet. Method C drops the unknowns and looks reasonable at 0.749, but it is also biased: the immediate failures, whose outcome is known the day it happens, enter the denominator at once, while the successes enter only after seven days, so the youngest cohort is over-represented by its failures. Method D takes only conversations that started at least seven days ago and reads their outcomes as of today. It reports 0.795 on 9,017 conversations, close to the truth, and it is the only one of the four that is a statement about a defined population. The price is delay: the number describes the release as it was a week ago.

The practical rules are to define cohorts by start time, to report the cohort date range with every number, and to compare arms at equal maturity. Early indicators from the lower rungs fill the waiting time. They can put a rollout on hold, and they cannot declare a win, because their relationship to the matured outcome has to be checked, which is the same point the article on monitoring, drift, and retraining makes about label maturity for classical models.

Outcomes the system of record cannot supply

Some properties have no system of record: whether the stated reason was supported by the policy, whether a refusal was justified, whether the tone fit the situation. These need an audit, meaning a probability sample of production sessions that a reviewer, or a model judge validated as in the earlier article, labels. The sampling design is the one used for the representative slice. The frame is a log of sessions, strata are the request types, rare and expensive strata get a floor of cases, and each case keeps its sampling probability so that estimates are weighted back to traffic. The audited sessions come from the content tier. They are redacted before review, kept for a short, stated period, and deleted by subject key on request, as the privacy article sets out. An audit sample is also the only ground truth against which the feedback signals in the next section can be calibrated.

For agent workflows the outcome is computed from the ledger of what the system did, as the article on evaluating agent actions and workflow outcomes does offline, and online it is a query over the same records: refunds issued that match a valid return, orders changed that the customer asked for. Human review steps produce their own measurements. The approval, reversal, and expiry rates from the decision audit in the article on human approval and escalation are outcome signals about the model’s proposals, with the same caution that article gives: an approved action is not necessarily a correct one, since reviewers who see little evidence or too many requests approve by habit.

Feedback and what it measures

Feedback is information that users or operators produce about an output. Three kinds are common in a language-model application. Explicit feedback is a thumbs control, a rating, or a free-text comment. Implicit feedback is behavior: rephrasing the same request, copying the answer, accepting a suggested action, abandoning the conversation, or asking for a person. Operator feedback is what staff do with the output: edits to a drafted reply, a rejection in an approval queue, a correction after the fact. All three are cheap to collect at scale compared with audits, and all three are produced by a process that selects which sessions generate a signal.

A signal is the product of an outcome and a response mechanism. Let \(\pi\) be the share of sessions that failed, \(d_f\) the probability that a customer in a failed session gives a thumbs-down, and \(d_o\) the probability that a customer in a fine session does. The thumbs-down rate per session is then

\[ q = \pi\, d_f + (1-\pi)\, d_o . \]

In words: the observed rate adds the failures that were reported and the successes that were reported anyway. The comparison of two releases by \(q\) is a comparison of \(\pi\) only if \(d_f\) and \(d_o\) are the same for both releases and over time. They are not guaranteed to be, because they depend on the interface that offers the control, on when it appears, on how long the answer is, and on who the customers are.

The ways a feedback signal misleads

Bias	Mechanism	Example	What to do
Self-selection and nonresponse	The decision to respond depends on the outcome or on traits that predict it	Angry customers rate; customers with a quick fix leave silently	Treat the response rate as data; compare it by release and segment; see the entry on selection and nonresponse bias
Presentation	The collection interface differs between arms or over time	The candidate’s layout moves the thumbs control below the fold	Make the collection mechanism part of the control; keep it identical across arms
Satisfaction without verification	Users judge what they can check	A customer told that a return is approved is satisfied until the refund fails	Pair feedback with the later outcome; audit
Click and acceptance bias	Accepting or clicking is influenced by position and default, beyond correctness	An agent accepts the first suggested reply because it is first	Use relative comparisons where possible; keep order and defaults identical
Ambiguous silence	No reaction can mean success or giving up	A customer who stops replying may be done or may be defeated	Add outcome measures that see the end state
Feedback loops	Collected data reflects the previous policy, and training on it changes the next	Retraining on accepted replies reinforces what the earlier model made easy to accept	Keep a randomized or audited slice outside the loop
Gaming and noise	Ratings can be coordinated or careless	A burst of identical comments on one release	Rate-limit, deduplicate, and weigh by a trusted audit

Click behavior has been studied as feedback for a long time. Joachims and colleagues compared the clicks on web search results with manual relevance judgments and reported that clicks are informative but biased by presentation, and that relative preferences derived from them were reasonably accurate on average. The same shape of result recurs here: absolute levels of a behavioral signal are hard to interpret, and comparisons made under identical presentation are more reliable. Public preference platforms collect votes in a similar way: Chiang and colleagues describe Chatbot Arena, which gathers pairwise votes on model replies from a crowdsourced user base. Such votes rank models in general, and a self-selected crowd is not a sample of one application’s traffic. Feedback is also used for learning. Hancock and colleagues describe a chatbot that extracts training examples from conversations after deployment and asks for feedback when it is uncertain about a reply. A system that learns from feedback closes the loop that the table lists, so the evaluation of such a system needs a sample that the loop does not touch.

A thumbs-down rate that falls while failures rise

The program below simulates 20,000 sessions for each of two releases. Release A fails in 10% of sessions and release B, the candidate, in 13%, so B is worse. A thumbs-down is given in 30% of A’s failed sessions. The candidate’s layout makes the control less prominent, so in B only 15% of failed sessions produce one. In fine sessions 1% of customers press thumbs-down in both. A reviewer also labels 400 randomly sampled sessions per release, independent of any feedback.

import math, random

rng = random.Random(12)
N = 20000          # sessions per release
AUDIT = 400        # sessions per release that are sampled at random and labeled by a reviewer

# configured behavior of the hypothetical releases and their feedback widgets
RELEASES = {
    # name: (true failure rate, P(thumbs-down | failed), P(thumbs-down | fine))
    "A (current)":   (0.10, 0.30, 0.010),
    "B (candidate)": (0.13, 0.15, 0.010),   # truly worse; the new layout makes the widget less prominent
}

def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h

results = {}
for name, (pi, d_fail, d_ok) in RELEASES.items():
    failed = [rng.random() < pi for _ in range(N)]
    down = [rng.random() < (d_fail if f else d_ok) for f in failed]
    audit_idx = rng.sample(range(N), AUDIT)
    audit_fail = sum(failed[i] for i in audit_idx)
    results[name] = (sum(failed) / N, sum(down) / N, audit_fail / AUDIT, wilson(audit_fail, AUDIT), (failed, down))
    print(f"{name:14s} true failure {sum(failed)/N:.3f} | thumbs-down per session {sum(down)/N:.4f} "
          f"| audited failure {audit_fail/AUDIT:.3f} Wilson [{wilson(audit_fail, AUDIT)[0]:.3f}, {wilson(audit_fail, AUDIT)[1]:.3f}]")

a, b = results["A (current)"], results["B (candidate)"]
print(f"\nthumbs-down rate, B relative to A: {b[1]/a[1]:.2f}  (looks like an improvement of {100*(1-b[1]/a[1]):.0f}%)")
print(f"true failure rate, B relative to A: {b[0]/a[0]:.2f}")

# what a thumbs-down actually tells you: precision and recall as a failure detector
for name in RELEASES:
    failed, down = results[name][4]
    tp = sum(f and d for f, d in zip(failed, down))
    print(f"{name:14s} thumbs-down recall of failures {tp/sum(failed):.2f}, "
          f"precision {tp/sum(down):.2f}, share of failures that never produce feedback {1-tp/sum(failed):.2f}")

# correcting with the audit: estimate P(down|fail) and P(down|fine) in the audited sample, then solve
for name in RELEASES:
    failed, down = results[name][4]
    rr = random.Random(5)
    idx = rr.sample(range(N), AUDIT)
    f_a = [i for i in idx if failed[i]]; ok_a = [i for i in idx if not failed[i]]
    se = sum(down[i] for i in f_a) / len(f_a); fp = sum(down[i] for i in ok_a) / len(ok_a)
    q = sum(down) / N
    est = (q - fp) / (se - fp)
    print(f"{name:14s} audit-based P(down|fail) {se:.2f}, P(down|fine) {fp:.3f}, "
          f"failure estimate from feedback volume {est:.3f}  (audit-only estimate {len(f_a)/AUDIT:.3f})")

# spread of the two estimators over repeated audits of the same 20,000 sessions
import statistics
print("\nrepeating the audit 1,000 times on the same sessions:")
for name in RELEASES:
    failed, down = results[name][4]
    q = sum(down) / N
    only, corr = [], []
    for _ in range(1000):
        idx = rng.sample(range(N), AUDIT)
        f_a = [i for i in idx if failed[i]]; ok_a = [i for i in idx if not failed[i]]
        only.append(len(f_a) / AUDIT)
        if f_a and ok_a:
            se = sum(down[i] for i in f_a) / len(f_a); fp = sum(down[i] for i in ok_a) / len(ok_a)
            if se - fp > 0.02:
                corr.append(min(1, max(0, (q - fp) / (se - fp))))
    print(f"{name:14s} true {sum(failed)/N:.3f} | audit only: mean {statistics.fmean(only):.3f} sd {statistics.stdev(only):.3f}"
          f" | feedback corrected by audit: mean {statistics.fmean(corr):.3f} sd {statistics.stdev(corr):.3f}")

The output:

A (current)    true failure 0.098 | thumbs-down per session 0.0386 | audited failure 0.080 Wilson [0.057, 0.111]
B (candidate)  true failure 0.128 | thumbs-down per session 0.0277 | audited failure 0.135 Wilson [0.105, 0.172]

thumbs-down rate, B relative to A: 0.72  (looks like an improvement of 28%)
true failure rate, B relative to A: 1.31
A (current)    thumbs-down recall of failures 0.31, precision 0.79, share of failures that never produce feedback 0.69
B (candidate)  thumbs-down recall of failures 0.14, precision 0.66, share of failures that never produce feedback 0.86
A (current)    audit-based P(down|fail) 0.37, P(down|fine) 0.006, failure estimate from feedback volume 0.091  (audit-only estimate 0.095)
B (candidate)  audit-based P(down|fail) 0.19, P(down|fine) 0.011, failure estimate from feedback volume 0.089  (audit-only estimate 0.130)

repeating the audit 1,000 times on the same sessions:
A (current)    true 0.098 | audit only: mean 0.098 sd 0.014 | feedback corrected by audit: mean 0.105 sd 0.035
B (candidate)  true 0.128 | audit only: mean 0.128 sd 0.016 | feedback corrected by audit: mean 0.149 sd 0.093

The raw feedback says the candidate is better. Thumbs-down per session falls from 0.0386 to 0.0277, a 28% drop, while the true failure rate rises by 31%. Nothing in the feedback stream is wrong in the sense of recording an event that did not happen. The response mechanism changed, and \(q\) moved with it. The detector view shows the mechanism: a thumbs-down recalls 31% of A’s failures and 14% of B’s, so 86% of the candidate’s failures never produce a signal. The precision is 0.79 and 0.66, which means a thumbs-down is a good pointer to a session worth reading, and the sessions it points to are mostly real failures, as long as the reviewer reads them.

The audit sample gets the direction right: 0.135 for B against 0.080 for A in this draw, with Wilson intervals of [0.105, 0.172] and [0.057, 0.111] that overlap only slightly. This draw overstates the gap, since the true rates are 0.098 and 0.128, and the intervals say how much a 400-session audit can be trusted. The two corrected estimates show why feedback does not replace the audit. If \(d_f\) and \(d_o\) are estimated from the audited sample, then \(q\) can be inverted for \(\pi\), as in \(\hat\pi = (q – \hat d_o)/(\hat d_f – \hat d_o)\), which is the same correction that appears for an imperfect judge. In this run it gives 0.091 for A and 0.089 for B, which is wrong for B because \(\hat d_f\) comes from only about 50 failed audited sessions. Repeating the audit 1,000 times on the same sessions shows the spread: the audit alone has a standard deviation of 0.014 and 0.016, and the feedback estimate corrected by the audit has 0.035 and 0.093, and is biased upward for B. The corrected estimate uses the same audit and adds noise. When a random audit exists, its estimate of the failure rate is the one to report, and the feedback stream is used to choose which sessions to read, to watch trends within one unchanged interface, and to raise alerts between audits.

The inversion also relies on an assumption that the simulation satisfies by construction: the response probabilities depend only on whether the session failed. Real response behavior also depends on the customer, the topic, and the time of day. An estimate that corrects for response then needs those variables in the model, and it is no more trustworthy than the assumption that nothing else drives response.

Collecting feedback without creating a privacy liability

A free-text comment often quotes the conversation, so the feedback store holds personal data in exactly the way the request log does. It belongs in the inventory of stores with an owner, a lifetime, and a deletion path keyed by subject, and a customer’s notice should say that feedback is kept and for what. The glossary entry on data retention covers lifetimes in general. Feedback that is copied into a test set or a fine-tuning set is a second copy that deletion has to reach, which is the “copies made by hand” row in the privacy article’s inventory. The structured fields, such as the release fingerprint, the thumbs value, the stratum, and the outcome, carry the evaluation and belong in the metadata tier. The text is kept in the content tier, redacted, for the shortest period that serves the review.

Staged exposure

Each stage of a rollout buys evidence of a different kind at a bounded cost to customers. The offline gate screens the candidate before any customer sees it, and the later stages move from evidence that needs no customer at all to evidence that needs many. Kohavi, Tang, and Xu give the general machinery of online controlled experiments, and the stages below add what is specific to a model, a prompt, and an index.

Stage	Exposure	Customers affected	Evidence it can give	What it cannot show
Offline gate	None	None	Regressions on the test set; behavioral tests	Live inputs, reactions, delayed outcomes
Shadow	Copies of live inputs go to the candidate; its outputs are not used	None	Behavior on live inputs; schema failures; latency and cost; disagreement with the current release	Customer reaction; effects of its actions
Internal use	Staff use the candidate in real tasks	Staff	Qualitative failures; usability	Representative traffic
Canary	A small, sticky share of customers (for example 1 to 5%)	Few	Gross harm; guardrail breaches; operational problems	Small differences in outcomes
Ramp	25%, then 50%	Many	Effect on the primary outcome against a concurrent control	Long-run effects
Full with holdback	All but a small holdback on the previous release	All	Time-varying effects; drift; novelty wearing off	Nothing that a holdback cannot show

The glossary entries on shadow testing and canary deployment describe the first and fourth stage in general. Two details apply to language-model applications. A shadow run copies an input, and copying must not copy an action: the shadow path calls tools in a mocked or read-only mode, because a shadow that submits a return creates a second real return. And a canary in the sense of release engineering is aimed at detecting harm from a small exposure. The Site Reliability Workbook chapter on canarying releases says that the canary population must be large enough and last long enough to be representative, and that its metrics should be tied to problems users actually have. The small exposure that detects gross harm is too small to detect a modest outcome difference, as the sample sizes below show, so a canary and an experiment are different instruments even when they use the same traffic split.

Assignment

A stage needs a rule that decides which customers get the candidate. The rule has to be stable, so that a customer’s later requests see the same release, it has to be monotone as exposure grows, so that customers in the 5% stage stay in the 25% stage, and it has to be independent between experiments that run at once. A hash of the experiment salt and the customer id does all three. The unit is the customer or the conversation, not the request, for the reasons in the article on experiment design: a conversation whose turns are served by different releases is not a measurement of either. The program below implements the rule, checks the properties, and shows two failures that the check is there to catch.

import hashlib, math, random
from collections import Counter

def bucket(unit_id: str, salt: str) -> int:
    """Stable bucket in 0..9999 for a unit (here a customer id) and an experiment salt."""
    h = hashlib.sha256(f"{salt}:{unit_id}".encode()).digest()
    return int.from_bytes(h[:8], "big") % 10000

def arm(unit_id, salt, exposure_pct):
    """Candidate if the unit's bucket is below the exposure; ramping up only adds units."""
    return "candidate" if bucket(unit_id, salt) < exposure_pct * 100 else "control"

customers = [f"cu_{i}" for i in range(50000)]
salt = "returns-prompt-v14"
prev = None
print("ramp: exposure -> candidate units, and how many earlier candidate units were dropped")
for pct in (1, 5, 25, 50):
    cand = {c for c in customers if arm(c, salt, pct) == "candidate"}
    dropped = len(prev - cand) if prev is not None else 0
    print(f"  {pct:3d}%  candidate {len(cand):6d} ({100*len(cand)/len(customers):5.2f}%)  dropped from previous stage: {dropped}")
    prev = cand

# a different salt gives an unrelated assignment, so two experiments do not share their treated users
other = {c for c in customers if arm(c, "returns-index-v7", 50) == "candidate"}
both = len(other & prev)
print(f"overlap of a 50% candidate group under another salt: {both} of {len(prev)} "
      f"(independent assignment would give about {len(prev)//2})")

def srm_p(n_cand, n_ctrl, planned_cand_share):
    """Chi-square test (1 df) of the observed split against the planned split."""
    n = n_cand + n_ctrl
    e1, e0 = n * planned_cand_share, n * (1 - planned_cand_share)
    chi2 = (n_cand - e1) ** 2 / e1 + (n_ctrl - e0) ** 2 / e0
    return chi2, math.erfc(math.sqrt(chi2 / 2))

# sessions logged per arm over a day: 5% planned. Case 1: healthy. Case 2: the candidate path
# crashes before logging for 8% of its sessions, so those sessions silently vanish.
rng = random.Random(3)
sessions = 40000
planned = 0.05
c_all = sum(rng.random() < planned for _ in range(sessions))
healthy = (c_all, sessions - c_all)
lost = (sum(rng.random() > 0.08 for _ in range(c_all)), sessions - c_all)
for name, (nc, nk) in (("healthy", healthy), ("8% of candidate sessions lost", lost)):
    chi2, p = srm_p(nc, nk, planned)
    print(f"{name:30s} candidate {nc:5d} control {nk:6d} share {nc/(nc+nk):.4f} chi2 {chi2:6.2f} p = {p:.4f}")

# assignment per request instead of per customer: one conversation sees both versions
requests_per_customer = 4
mixed = 0
for c in customers[:5000]:
    arms = {arm(f"{c}:{k}", salt, 50) for k in range(requests_per_customer)}
    mixed += len(arms) == 2
print(f"per-request assignment at 50%: {mixed} of 5000 customers saw both versions "
      f"(expected {5000*(1-2*0.5**requests_per_customer):.0f})")

The output:

ramp: exposure -> candidate units, and how many earlier candidate units were dropped
    1%  candidate    516 ( 1.03%)  dropped from previous stage: 0
    5%  candidate   2536 ( 5.07%)  dropped from previous stage: 0
   25%  candidate  12445 (24.89%)  dropped from previous stage: 0
   50%  candidate  24985 (49.97%)  dropped from previous stage: 0
overlap of a 50% candidate group under another salt: 12566 of 24985 (independent assignment would give about 12492)
healthy                        candidate  1989 control  38011 share 0.0497 chi2   0.06 p = 0.8008
8% of candidate sessions lost  candidate  1834 control  38011 share 0.0460 chi2  13.23 p = 0.0003
per-request assignment at 50%: 4400 of 5000 customers saw both versions (expected 4375)

The ramp lines show the monotone property: the candidate group grows from 516 to 24,985 customers and no customer from an earlier stage is dropped. The group sizes follow the intended percentages to within sampling variation. A second experiment with a different salt overlaps the first in 12,566 of 24,985 customers, close to the 12,492 that independence predicts, so two concurrent experiments do not share their treated customers by accident.

The sample ratio check detects a failure that no outcome metric shows. The planned split is 5% candidate, and a healthy day logs 1,989 candidate sessions against 38,011 control, which agrees with the plan (chi-square 0.06, p = 0.80). In the second case the candidate path crashes before logging for 8% of its sessions. The observed share is 0.0460, the chi-square is 13.23, and p is 0.0003. Without this check, the sessions that crashed would be missing from the candidate’s metrics, the candidate’s success rate would be computed over sessions that survived, and the candidate would look better than it is. A mismatch between planned and observed arm sizes is therefore the first thing to test on every stage, as the experiment-design article recommends. The last line shows why the unit is not the request: with four requests per customer and per-request assignment at 50%, 4,400 of 5,000 customers saw both versions.

Interference that is specific to these applications

Random assignment gives comparable groups only if the arms do not affect each other. Several components of an AI application are shared by default. A response cache keyed by the normalized query serves a candidate-produced answer to a control customer, and the reverse, unless the key includes the release fingerprint, and the privacy article already requires that caches not cross principals. A shared retrieval index serves both arms the same evidence, which is correct when the index is not the thing being changed and hides the effect when it is, so an index change needs one index version per arm, selected by alias as in the article on building a retrieval index. Conversation memory written under the candidate is read later under the control if the customer is reassigned. And capacity is shared: a candidate with longer prompts queues behind the control or ahead of it, so latency in one arm depends on the other. Logging the release fingerprint of every request, as the article on context and prompt interfaces describes, makes it possible to check which release produced each cache entry, memory note, and answer.

How much traffic a stage needs

The length of a stage follows from the question it asks. Two sample-size calculations cover most of the decisions. The first is about rare, severe failures. If a failure has probability \(p\) per session and sessions are independent, then the chance of seeing at least one in \(n\) sessions is \(1-(1-p)^n\), so seeing one with 95% probability needs \(n = \ln 0.05 / \ln(1-p) \approx 3/p\). The same relation read backwards is the rule of three, described by Hanley and Lippman-Hand: after \(n\) sessions with no failure, an approximate 95% upper bound on \(p\) is \(3/n\), and the exact bound is \(1 – 0.05^{1/n}\). The second calculation is the number of sessions per arm needed to detect a change from \(p_0\) to \(p_1\) with a two-sided level \(\alpha\) and power \(1-\beta\) by a two-proportion z-test:

\[ n = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_0(1-p_0)+p_1(1-p_1)}\right)^2}{(p_1-p_0)^2}, \qquad \bar p = \frac{p_0+p_1}{2}. \]

Here \(z_q\) is the standard normal quantile. The formula is the normal approximation for independent sessions, and it assumes one session per customer or that clustering is handled as in the experiment-design article.

import math
from statistics import NormalDist

Z = NormalDist().inv_cdf

def n_to_see_one(p, conf=0.95):
    """Sessions needed so that P(at least one failure) >= conf when the failure rate is p."""
    return math.ceil(math.log(1 - conf) / math.log(1 - p))

def upper_bound_zero(n, conf=0.95):
    """Exact one-sided upper bound on the rate after n sessions with zero failures: 1 - (1-conf)^(1/n)."""
    return 1 - (1 - conf) ** (1 / n)

def n_per_arm(p0, p1, alpha=0.05, power=0.80):
    """Two-sided two-proportion z-test, equal arms (normal approximation)."""
    pb = (p0 + p1) / 2
    num = (Z(1 - alpha / 2) * math.sqrt(2 * pb * (1 - pb)) + Z(power) * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
    return math.ceil(num / (p1 - p0) ** 2)

print("severe failure (rate p): sessions needed to see at least one with 95% probability")
for p in (0.01, 0.005, 0.001):
    print(f"  p = {p:.3f}: {n_to_see_one(p):6d} sessions   (3/p = {3/p:.0f})")

print("\nafter n candidate sessions with zero severe failures, 95% upper bound on the rate")
for n in (100, 500, 2000, 10000):
    print(f"  n = {n:6d}: exact {upper_bound_zero(n):.5f}   rule of three {3/n:.5f}")

print("\nsessions per arm to detect a drop in task success at 80% power, alpha 0.05 (two-sided)")
for p0, p1 in ((0.80, 0.78), (0.80, 0.75), (0.80, 0.70)):
    n = n_per_arm(p0, p1)
    print(f"  {p0:.2f} -> {p1:.2f}: {n:6d} per arm")

DAILY = 4000   # hypothetical eligible sessions per day
print(f"\ndays to collect the 0.80 -> 0.78 sample at {DAILY} sessions/day, by candidate exposure")
need = n_per_arm(0.80, 0.78)
for pct in (5, 25, 50):
    cand_per_day = DAILY * pct / 100
    ctrl_per_day = DAILY - cand_per_day
    days = math.ceil(need / min(cand_per_day, ctrl_per_day))
    print(f"  {pct:2d}% exposure: candidate {cand_per_day:5.0f}/day, control {ctrl_per_day:5.0f}/day -> {days:3d} days (equal-arm formula, smaller arm limits)")

The output:

severe failure (rate p): sessions needed to see at least one with 95% probability
  p = 0.010:    299 sessions   (3/p = 300)
  p = 0.005:    598 sessions   (3/p = 600)
  p = 0.001:   2995 sessions   (3/p = 3000)

after n candidate sessions with zero severe failures, 95% upper bound on the rate
  n =    100: exact 0.02951   rule of three 0.03000
  n =    500: exact 0.00597   rule of three 0.00600
  n =   2000: exact 0.00150   rule of three 0.00150
  n =  10000: exact 0.00030   rule of three 0.00030

sessions per arm to detect a drop in task success at 80% power, alpha 0.05 (two-sided)
  0.80 -> 0.78:   6510 per arm
  0.80 -> 0.75:   1094 per arm
  0.80 -> 0.70:    294 per arm

days to collect the 0.80 -> 0.78 sample at 4000 sessions/day, by candidate exposure
   5% exposure: candidate   200/day, control  3800/day ->  33 days (equal-arm formula, smaller arm limits)
  25% exposure: candidate  1000/day, control  3000/day ->   7 days (equal-arm formula, smaller arm limits)
  50% exposure: candidate  2000/day, control  2000/day ->   4 days (equal-arm formula, smaller arm limits)

Seeing a failure that happens once in a thousand sessions takes about 3,000 candidate sessions, and a stage that has served 2,000 candidate sessions with no such failure has only shown that the rate is below 0.0015 with 95% confidence. At a 5% canary on 4,000 sessions a day, 200 candidate sessions a day, the 2,000 sessions take ten days. Detecting a drop in task success from 0.80 to 0.78 takes 6,510 sessions per arm, which at 5% exposure is 33 days of candidate traffic and at 50% exposure is 4 days. The days column uses the smaller arm and is therefore conservative for unequal arms. On top of that, a seven-day outcome adds seven days of maturity to every stage before its result can be read.

Two design conclusions follow. A canary is sized to detect harm that is large or frequent, and its stop rules are written for that. A ramp is sized, in days, for the smallest outcome change that would change the decision, and if the required sample is out of reach, the choices are to accept a larger minimum detectable effect, to expose more traffic, or to lean on audits, which measure the outcome directly on a smaller number of sessions. The stage plan comes from these calculations, not from calendar habit.

Repeated looks at a running stage

A rollout is monitored continuously, and every look at the data is a chance to see a difference that is only noise. The article on experiment design shows with a simulation that repeating a fixed 5% test at ten looks inflates the false-positive rate to about 21% and describes group-sequential and always-valid methods for planned interim decisions. The program below applies the lesson to a rollback rule. The current release has a severe wrong-decision rate of 1.0%. A rollout runs seven days at 4,000 eligible sessions a day, with candidate exposure 5%, 5%, 25%, 25%, 50%, 50%, 50% and one planned look per day on the cumulative counts. The rule rolls back when a one-sided pooled z-test of candidate against control, with at least five candidate events, exceeds the threshold. It compares a naive level of 0.05 at every look with a Bonferroni split, \(0.05/7\) per look.

import math, random, statistics
from statistics import NormalDist

DAILY = 4000                                   # hypothetical eligible sessions per day
SCHEDULE = [0.05, 0.05, 0.25, 0.25, 0.50, 0.50, 0.50]    # candidate exposure on days 1..7
LOOKS = len(SCHEDULE)                          # one planned look per day
BASE = 0.010                                   # severe wrong-decision rate of the current release
Z1 = NormalDist().inv_cdf

def one_sided_z(xc, nc, xk, nk):
    """z statistic for H1: candidate rate > control rate (pooled variance)."""
    p = (xc + xk) / (nc + nk)
    if p == 0 or p == 1:
        return 0.0
    se = math.sqrt(p * (1 - p) * (1 / nc + 1 / nk))
    return (xc / nc - xk / nk) / se

def decide(xc, nc, xk, nk, alpha_look, min_events=5):
    if xc < min_events:
        return "continue"
    return "ROLL BACK" if one_sided_z(xc, nc, xk, nk) > Z1(1 - alpha_look) else "continue"

def simulate(cand_rate, alpha_look, rng):
    xc = nc = xk = nk = 0
    for day, e in enumerate(SCHEDULE, start=1):
        n_c = round(DAILY * e); n_k = DAILY - n_c
        xc += rng.binomialvariate(n_c, cand_rate); nc += n_c
        xk += rng.binomialvariate(n_k, BASE); nk += n_k
        if decide(xc, nc, xk, nk, alpha_look) == "ROLL BACK":
            return day, nc
    return None, nc

REPS = 4000
rules = {"naive: alpha 0.05 at every look": 0.05, f"corrected: 0.05/{LOOKS} per look": 0.05 / LOOKS}
for truth, cand in (("candidate equal to control (1.0%)", 0.010), ("candidate worse (1.5%)", 0.015), ("candidate worse (2.0%)", 0.020)):
    print(truth)
    for name, a in rules.items():
        rng = random.Random(21)
        days, exposed = [], []
        for _ in range(REPS):
            d, n = simulate(cand, a, rng)
            days.append(d); exposed.append(n)
        stopped = [d for d in days if d is not None]
        med = statistics.median(stopped) if stopped else float("nan")
        print(f"  {name:36s} rolled back {100*len(stopped)/REPS:5.1f}% of runs"
              f" | median day {med:3.1f} | mean candidate sessions served {statistics.fmean(exposed):7.0f}")

The output:

candidate equal to control (1.0%)
  naive: alpha 0.05 at every look      rolled back  18.1% of runs | median day 3.0 | mean candidate sessions served    7240
  corrected: 0.05/7 per look           rolled back   5.0% of runs | median day 3.0 | mean candidate sessions served    8075
candidate worse (1.5%)
  naive: alpha 0.05 at every look      rolled back  98.3% of runs | median day 3.0 | mean candidate sessions served    2242
  corrected: 0.05/7 per look           rolled back  89.7% of runs | median day 4.0 | mean candidate sessions served    3956
candidate worse (2.0%)
  naive: alpha 0.05 at every look      rolled back 100.0% of runs | median day 2.0 | mean candidate sessions served     818
  corrected: 0.05/7 per look           rolled back 100.0% of runs | median day 3.0 | mean candidate sessions served    1309

When the candidate is equal to the control, the naive rule rolls back 18.1% of the runs, and the corrected rule 5.0%. A false rollback is not free: it discards a good release, delays the improvement, and teaches the team to override alerts. When the candidate is worse, the naive rule acts sooner and serves fewer candidate sessions: at a true 2.0% the median detection day is 2 against 3, and the mean number of candidate sessions served before stopping is 818 against 1,309. The correction costs a day and about 500 more candidate sessions in that case, and at a true 1.5% it detects 89.7% of the runs against 98.3%. Neither setting is correct in general. The two errors have different costs, a false rollback costs a delay and a missed detection costs harm to customers, so the level of a guardrail is chosen from those costs, written down, and applied as planned instead of being chosen at the moment a number looks bad. The pooled counts also assume that the true rates are constant across days. When exposure changes and rates drift with the day of the week, the comparison should be made within each day and combined, and not pooled across days. Adaptive allocation, in which a bandit moves traffic toward the apparently better release, changes how many sessions each arm gets and complicates the effect estimate, as the article on bandits and exploration shows.

Rollback criteria

A rollback criterion is a condition, fixed before exposure begins, together with the action that follows when it holds. The reason for fixing it in advance is the one given for acceptance criteria in the framing article: a threshold chosen while looking at the data describes what happened and decides nothing. A criterion has a metric with its computation and window, a population, a threshold, a statistical rule for uncertainty, an action, and an owner who can execute the action without asking for permission at the time.

Four classes cover most criteria, and they are checked in this order.

• Hard stops are events for which one occurrence is enough: a confirmed disclosure of personal data to a party that is not entitled to it, an action executed that the dispatcher should have blocked. No statistics are involved, because the cost of waiting is larger than the cost of a false alarm.
• Guardrails protect outcomes that must not get worse while the release tries to improve another: the severe wrong-decision rate, the fallback rate, the escalation rate. They are evaluated with intervals, and the vocabulary is the one in the glossary entry on primary and guardrail metrics.
• The primary outcome is the matured task success rate against the concurrent control, with a margin that the team is willing to accept.
• Operational limits are latency percentiles, error rate, and cost per session.

A fourth action joins the usual two. Rolling back returns exposure to the previous release. Proceeding moves to the next stage. Holding keeps the current exposure and continues to collect, which is the correct response to a result that cannot decide, the third outcome that the framing article’s acceptance criteria already have. The program below encodes the criteria as a function of a stage’s counts and evaluates six hypothetical stages. The limits are examples chosen for the illustration: a severe-error limit of 2%, a task-success margin of three points below the control’s 0.795, and a latency limit of 3,000 ms.

import math
from dataclasses import dataclass

def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h

@dataclass
class Stage:
    candidate_sessions: int
    privacy_incidents: int          # confirmed disclosure to a party not entitled to it
    unauthorized_actions: int       # tool calls that the dispatcher should have blocked but executed
    severe_wrong_decisions: int     # audited sessions with a wrongly approved or denied return
    audited_sessions: int
    matured_sessions: int           # sessions old enough for the 7-day outcome
    successes: int                  # matured sessions with task success
    p95_latency_ms: float

CONTROL_SUCCESS = 0.795             # measured on the concurrent control, matured cohort
CRITERIA_NOTE = "hard stops > guardrails > primary outcome > operations"

def evaluate(s: Stage, min_audits=300, min_matured=1500):
    reasons = []
    # 1. hard stops: one event is enough, no statistics involved
    if s.privacy_incidents > 0:
        return "ROLL BACK", ["privacy incident count > 0"]
    if s.unauthorized_actions > 0:
        return "ROLL BACK", ["unauthorized action count > 0"]
    # 2. severe-error guardrail: roll back when the 95% lower bound is above the 2% limit; hold when the sample cannot show it is below
    if s.audited_sessions < min_audits:
        reasons.append(f"audits {s.audited_sessions} < {min_audits}")
        action = "HOLD"
    else:
        lo, hi = wilson(s.severe_wrong_decisions, s.audited_sessions)
        if lo > 0.02:
            return "ROLL BACK", [f"severe wrong-decision rate interval [{lo:.3f}, {hi:.3f}] is above the 0.020 limit"]
        if hi > 0.02:
            reasons.append(f"severe wrong-decision interval [{lo:.3f}, {hi:.3f}] still reaches the 0.020 limit")
            action = "HOLD"
        else:
            action = "PROCEED"
    # 3. primary outcome: only judged on matured sessions
    if s.matured_sessions < min_matured:
        reasons.append(f"matured sessions {s.matured_sessions} < {min_matured}")
        action = "HOLD"
    else:
        lo, hi = wilson(s.successes, s.matured_sessions)
        if hi < CONTROL_SUCCESS - 0.03:
            return "ROLL BACK", [f"task success interval [{lo:.3f}, {hi:.3f}] is entirely below control minus 0.03"]
        if lo < CONTROL_SUCCESS - 0.03:
            reasons.append(f"task success interval [{lo:.3f}, {hi:.3f}] reaches below control minus 0.03")
            action = "HOLD"
    # 4. operations
    if s.p95_latency_ms > 3000:
        return "ROLL BACK", [f"p95 latency {s.p95_latency_ms:.0f} ms > 3000 ms"]
    return action, reasons or ["all criteria met for this stage"]

cases = {
    "A: 5% stage, early": Stage(1800, 0, 0, 2, 250, 600, 475, 2100),
    "B: 25% stage, clean": Stage(8000, 0, 0, 15, 1500, 4200, 3330, 2300),
    "C: 25% stage, severe errors": Stage(8000, 0, 0, 45, 1500, 4200, 3330, 2300),
    "D: 25% stage, borderline": Stage(8000, 0, 0, 24, 1500, 4200, 3330, 2300),
    "E: 25% stage, outcome drop": Stage(8000, 0, 0, 15, 1500, 4200, 3000, 2300),
    "F: one privacy incident": Stage(8000, 1, 0, 15, 1500, 4200, 3330, 2300),
}
for name, st in cases.items():
    act, why = evaluate(st)
    print(f"{name:28s} -> {act}")
    for w in why:
        print(f"    {w}")

The output:

A: 5% stage, early           -> HOLD
    audits 250 < 300
    matured sessions 600 < 1500
B: 25% stage, clean          -> PROCEED
    all criteria met for this stage
C: 25% stage, severe errors  -> ROLL BACK
    severe wrong-decision rate interval [0.022, 0.040] is above the 0.020 limit
D: 25% stage, borderline     -> HOLD
    severe wrong-decision interval [0.011, 0.024] still reaches the 0.020 limit
E: 25% stage, outcome drop   -> ROLL BACK
    task success interval [0.700, 0.728] is entirely below control minus 0.03
F: one privacy incident      -> ROLL BACK
    privacy incident count > 0

Scenario A is a 5% stage with 250 audits and 600 matured sessions. The result is a hold because neither sample is large enough, and the function says which. Scenario B, with 15 severe errors in 1,500 audits, proceeds, because the upper end of the Wilson interval stays below the 2% limit. Scenario C, with 45 severe errors in the same number of audits, rolls back because the whole interval, 0.022 to 0.040, is above the limit. Scenario D has an observed rate of 1.6%, which is under the limit, and holds because the interval reaches 0.024: the sample cannot show that the rate is below the limit, and neither can it show the opposite. Scenario E has acceptable severe errors and a matured task success of 0.714 against a floor of 0.765, an interval entirely below the floor, and rolls back. Scenario F has acceptable numbers and one privacy incident, and rolls back on the hard stop. The order in the function is part of the design. A hard stop is evaluated before anything else so that no interval can outvote it, and a hold from an unmet sample size is returned only when nothing worse has been found.

What a rollback restores

The unit that rolls back is the release, which is the fingerprint of everything that determines behavior: the prompt, the model identifier and decoding parameters, the retrieval index version, the tool contracts and validators, and the policy and threshold configuration. Rolling back one component to its old value restores the old release only if the others are compatible with it. The table lists, for the changes this article has used, what each is coupled to and what a rollback of it needs.

Change	Coupled to	Extra evidence during rollout	A rollback needs
Prompt	Output schema, tool contracts, behavioral tests	Schema-valid rate, behavioral suite on live samples	The old prompt still matches the current schema and tool contract
Model	Formatting, refusals, latency, cost, tokenization	Latency and cost percentiles, refusal rate, audits	The old model ID still available; its retirement date is known
Index or corpus	Chunking, embedding recipe, cached answers and embeddings	Retrieval metrics on a sample; citation validity	The previous index version is retained until the window ends; caches keyed to the index
Tool or policy configuration	Side effects, approval thresholds	Action ledger reconciliation; approval and reversal rates	Compensation for actions already taken
Provider-side change with no change on your side	Behavior behind an unchanged identifier	Scheduled replay of an anchor set; canary probes	A different route or the baseline, since the provider’s change cannot be reverted by you

Pinned versions matter here. Anthropic’s documentation on model IDs and versioning, as one example, states that a model ID identifies a pinned version whose weights do not change, that aliases accepted for some earlier models are convenience pointers to the most recent dated snapshot, that each ID has its own retirement schedule, and that the serving infrastructure around a fixed model can change and occasionally alter observable behavior. Other providers differ in how they name and retire versions, so the application records the exact identifier it called, as the fingerprint requires. Two consequences are worth planning for. A rollback target that has been retired is not a target, so retirement dates enter the release record. And a behavior change behind an unchanged identifier cannot be seen by comparing releases, because nothing in your release changed. It is seen by a scheduled replay of the anchor set from the reproducible-evaluation article against production, with an alert when a score leaves its recorded range, and by the model API integration practices for provider changes and tests. The retrieval side has its own online checks: the article on evaluating retrieval and answers separately supplies recall and groundedness measures that can be computed on audited production samples after an index change.

Executing the rollback

A rollback that has never been performed is a hypothesis. The switch is a configuration change in the routing layer, such as a flag or a routing table entry, and not a redeploy, so that it takes effect in minutes. It is exercised before the first stage, and the exercise has a measurable result: after the switch, the share of requests whose logged fingerprint is the candidate’s falls to zero within a stated time. That check also catches the case where an alias was repointed but a long-running process still holds the old model, which the glossary entry on rollback warns about. Sessions that began under the candidate either finish under it, when the release is merely suspect, or restart on the control, when the release is known to be harmful, and the rule is chosen in advance.

Rollback restores code and configuration. It does not restore what the candidate produced. Conversation summaries, memory notes, cache entries, and index entries written while the candidate was live remain, so each carries the fingerprint that created it, and the rollback includes a step that purges or quarantines those with the candidate’s fingerprint. Actions the candidate took in the world, such as a refund issued or a message sent, are not undone by a switch, and the correction of each is the compensation described in the article on durable agent workflows. Approval requests that the candidate created while waiting are in the queue under a stale basis and are re-evaluated or expired. When the previous release is itself not safe, because a policy changed or the old index contains withdrawn documents, the correct response may be a forward fix, or a switch to the non-AI baseline from the framing article, which doubles as the fallback.

After a rollback, the failed sessions are the most valuable data in the system. They become regression cases for the test set, with the scrubbing, retention, and subject-key deletion that the privacy article requires of anything copied out of the logs, and the stage that missed the problem gets a note: which criterion would have caught it earlier, and with what sample.

What online evaluation does not settle

A rollout that passes every stage has shown that the candidate did not do harm that the measurements could see, in the period and the traffic that were measured. Several things remain open. A change in how customers behave can take weeks to appear, and a novelty effect can inflate early numbers, which is the reason for a holdback that stays on the previous release for a while after full rollout. Segments with little traffic, such as a language or a product line, are underpowered in every stage, and they need their own audit plan, not a share of the global one. A seasonal period or a policy change can make a comparison between a stage in November and a stage in December meaningless, so a comparison is always against a concurrent control. High-stakes tasks, where an error exposes a customer to financial or legal harm, are not appropriate for learning by exposure: they get shadow runs, audits of the candidate’s proposed actions, and human approval of the actions before any customer is exposed, as the article on human approval describes. And the question of what an evaluated improvement costs, in tokens, latency, and review time, belongs to the next article, optimizing AI applications for quality, latency, and cost, which builds on the measurements and the stage discipline described here.

References

• Kohavi, R., Tang, D., Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. https://experimentguide.com/
• Google. Site Reliability Engineering Workbook, Chapter 16: Canarying Releases. https://sre.google/workbook/canarying-releases/
• Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G. (2005). Accurately Interpreting Clickthrough Data as Implicit Feedback. Proceedings of SIGIR 2005. https://pure.psu.edu/en/publications/accurately-interpreting-clickthrough-data-as-implicit-feedback/
• Hancock, B., Bordes, A., Mazaré, P.-E., Weston, J. (2019). Learning from Dialogue after Deployment: Feed Yourself, Chatbot! arXiv:1901.05415. https://arxiv.org/abs/1901.05415
• Chiang, W.-L., et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132. https://arxiv.org/abs/2403.04132
• Hanley, J. A., Lippman-Hand, A. (1983). If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators. JAMA 249(13), 1743-1745. Summary: https://en.wikipedia.org/wiki/Rule_of_three_(statistics)
• Anthropic. Models overview, and Model IDs and versioning (Claude API documentation). https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions