AI Platform Incident Response and Recovery

Consider a fictional support assistant that answers customer questions about a return policy and, for some requests, issues refunds and sends confirmation emails through tools. On a Tuesday morning the team releases a new bundle: a new model version, an edited prompt, and a rebuilt retrieval index. Every dashboard stays green. No request fails, latency does not move, and the error rate is zero. Over the next half hour the assistant approves refunds that policy says a human must approve, and it tells customers that the return window is 60 days when it is 30.

A conventional incident response playbook starts from an alert about errors or latency, and neither alert would fire here. The service is healthy and the product is wrong. This article is about responding to that kind of failure: noticing it, stopping it from getting worse, restoring a known-good state, finding out what already happened, and proving that it is safe to resume. One distinction organizes the rest. Some effects of a bad release can be undone by changing configuration, and some cannot be undone by anything the platform does, because they already left the building as money, email, or a third party’s action. The platform’s job during an incident is to stop the second kind from growing while it reverses the first kind.

The article assumes you have read Controlling AI Cost per Successful Task, in particular the idea that a task is the unit of record and that retries, escalations, and wrong deliveries have costs of their own. It also builds on the artifact versioning, gateway, evaluation gate, agent runtime, observability, and service objective articles earlier in this course, such as versioning models, data, prompts, and evaluation artifacts and agent runtime isolation, tools, and execution policies. Basic operations knowledge (on-call, alerts, deployments) is enough to follow the rest.

The experiments are local simulations in Python 3.12.14 on macOS (Darwin 25.6.0), using only the standard library. The “assistant” is a set of rules that reproduces the fictional failure, not a language model, and the traffic, timings, and response delays are invented. They show how the incident machinery behaves, and they do not measure any real system.

Why AI incidents are quiet

Most service failures announce themselves through the transport: a connection fails, a status code is 500, a queue backs up. An AI feature adds failures that complete successfully. The model returns well-formed text, the tool call has valid arguments, the downstream API accepts it. What is wrong is the content or the decision, and nothing in the request path knows that. The simulated incident makes this concrete. Over a simulated eight-hour day with 1,449 requests, the log shows this about the release window.

requests: 1449   http_status != 200: 0
latency p95 ms: before release 957, after release 987
release at 02:00:00

There is not one non-200 status in the day, and the 95th-percentile latency moved from 957 to 987 milliseconds, which is within the noise of the lognormal latency the simulation draws. A service-level objective defined on availability and latency, the usual first two, would report that the release was fine. The earlier article on service objectives treats task success as an objective alongside latency and availability, and incident response is where that choice is tested.

Detection therefore needs signals that observe behavior and outcomes. They fall into four groups that differ in speed and in what they can see.

Signal group	Examples	Speed	Blind spot
Operational	Error rate, latency, saturation, provider status	Seconds to minutes	Content and decisions that complete normally
Behavioral	Tool-call mix, refusal rate, output length, refund amount per interval, retrieval hit rate	Minutes	Needs a baseline; shifts that match no metric you chose
Outcome	Task success on a canary set, sampled audits, user feedback, complaints, reopen rate	Minutes to days	Slow, and sampling can miss rare failures
Invariant	Policy checks at the tool boundary, such as a refund above the limit without an approval	Immediate	Only covers rules that someone encoded as checks

In the simulation the release happens at 02:00:00 on the simulated clock. The behavioral signal, total refund dollars executed per 15-minute window compared with a baseline of the eight windows before the release, crossed four standard deviations at the first window boundary after the release. The outcome signal, a 5 percent random sample of requests reviewed by a person with a 10-minute delay, found its first bad request later.

signal refund_dollars_z   first fired 02:15:00  (+15 min after release)
signal audit              first fired 02:24:19  (+24 min after release)

The refund-dollars signal fires first because the fault, extra large refunds, moves that metric directly. It would not have noticed the wrong return window at all, which affects text and not dollars. The audit signal covers both faults and is slower, because it needs a bad request to be sampled and then reviewed. The numbers depend on how large the effect is and on the sampling rate, and a subtler fault takes longer to detect. A set of signals chosen for the faults that already happened will miss the next kind, and that is the argument for the invariant row of the table, because a check at the tool boundary does not depend on guessing which metric will move.

Classifying and prioritizing

Kinds of AI incident

The response differs by what failed, so the first step is to name the kind. The kinds below overlap, and one event can be several at once.

• Bad output. The system produced wrong, harmful, or off-policy content, and no external action followed. The effect lives in what a user read, which cannot be recalled, but it stops when the cause stops.
• Unsafe action. An agent used a tool to cause an external effect it should not have caused: a payment, a message, a deletion, a call to another system. This is the most severe kind because the effect is outside the platform.
• Data exposure. One tenant’s or user’s data reached someone not entitled to it, through retrieval, caching, logs, or a tool. It is a security incident as covered in the earlier article on tenant isolation, and it is also an AI incident because the cause may be a prompt or an index.
• Provider failure. The model provider is unavailable, slow, rate-limiting, or returning degraded results, or has changed behavior or terms.
• Runaway cost. A loop, retry storm, or misrouted traffic spends the budget. The cost article’s step and spend caps are the containment, and cost is often the first signal of the other kinds.
• Upstream data fault. A corrupted or stale index, a bad evaluation set, or a broken document pipeline makes a correct system produce wrong answers.

Severity from reversibility, reach, and persistence

A severity scheme for AI incidents can use three questions. The first is reversibility: do the bad effects exist only inside the platform, where configuration can undo them, or have they left it? The second is reach: how many tenants, users, and requests are affected, and could the effect cross a boundary it should not? The third is persistence: is the cause still active, so the damage grows with every request? An incident with irreversible effects and active persistence is the highest severity even at low volume, because the cost of a delay is unbounded and cannot be recovered later. A bad-output incident with no actions and a cause that has been stopped can wait for working hours.

Two public frameworks give vocabulary for this. The NIST AI Risk Management Framework’s Manage function includes an outcome about mechanisms to supersede, disengage, or deactivate AI systems whose performance or outcomes are inconsistent with intended use. The Playbook’s suggested actions include setting thresholds for when bypass or deactivation is needed, keeping backup systems, preserving evidence, analyzing why deactivation happened, and defining criteria for safely returning a system to production. A second outcome covers communicating incidents and errors to relevant parties and keeping records of reported errors, incidents, and responses. Google’s site reliability engineering material adds the operational rule that first responders should prioritize mitigation over root cause, since knowing roughly where the problem lies is enough to act, and recommends preparing generic mitigations such as rollbacks before they are needed. The rest of this article turns those into concrete mechanisms.

Containment

The ladder of containment actions

Containment means stopping an incident from causing more harm before the cause is understood. The actions below run from least to most disruptive, and a good runbook uses the least disruptive one that is enough.

1. Disable specific tools. A feature flag used as a kill switch turns off one tool, such as issue_refund or send_email. The assistant keeps answering questions, but the effects that cannot be undone stop. This is usually the first move when an unsafe action is the concern.
2. Read-only mode. All write tools off, answers still served. A superset of the first step, for when the set of unsafe tools is unclear.
3. Degrade to a safe path. Return a fixed message, route to a human queue, or answer from a static page. Quality drops, but behavior is predictable.
4. Repoint to a known-good bundle. Change the alias so requests are served by the previous release.
5. Cut traffic. Block a tenant, a feature, or all traffic at the gateway. This stops everything, including the legitimate use, and is the last resort.

Two design choices decide whether these actions work when they are needed. The first is where the flag is checked. The policy that enables a tool must be evaluated at the moment the tool executes, in the tool executor, and not only when a request starts. An agent run that began before the flag changed can still be mid-plan, and a flag checked only at request admission would let it finish its refund. The second is that every containment action must be one that can be taken in one step by whoever is on call, without a deployment. Mitigations that need a code change are not containment.

The simulation implements both mechanisms. The gateway resolves the alias support-assistant to a release bundle, and each of three replicas caches that resolution for 300 seconds. The tool executor polls the flag store every 10 seconds.

class Gateway:
    """Each replica caches the alias for `ttl` seconds, so a repoint takes up to ttl to reach every replica."""
    def __init__(self, replicas=3, ttl=300):
        self.history = [(0, "r-56")]; self.ttl = ttl
        self.cached = ["r-56"] * replicas; self.expiry = [i * 100 for i in range(replicas)]
    def authoritative(self, t):
        return [b for ts, b in self.history if ts <= t][-1]
    def resolve(self, t, replica):
        if t >= self.expiry[replica]:
            self.cached[replica] = self.authoritative(t); self.expiry[replica] = t + self.ttl
        return self.cached[replica]

class Flags:
    """Tool executors poll the flag store every `poll` seconds."""
    def __init__(self, poll=10):
        self.history = [(0, "issue_refund", True), (0, "send_email", True)]; self.poll = poll
    def enabled(self, t, name):
        seen = (t // self.poll) * self.poll
        vals = [v for ts, n, v in self.history if n == name and ts <= seen]
        return vals[-1]

The bundle pins the artifacts that were released together, so that the model, prompt, and index cannot drift apart.

POLICY_LIMIT, RETURN_DAYS = 100, 30      # auto-refund only if amount <= 100 and inside the return window
BUNDLES = {                               # a release bundle pins model, prompt and retrieval index together
    "r-56": {"model": "m-v11", "prompt": "p-40", "index": "i-1030"},
    "r-57": {"model": "m-v12", "prompt": "p-41", "index": "i-1120"},
    "r-58": {"model": "m-v12", "prompt": "p-42", "index": "i-1130"},   # the later fix
}
GOOD_PROMPTS, GOOD_INDEXES = {"p-40", "p-42"}, {"i-1030", "i-1130"}

In the timeline the simulation uses, an engineer acknowledges the alert and flips the tool flags 10 minutes after it fires, and repoints the alias 25 minutes after it fires. These delays are assumptions, set so the sequence is visible. They are not measurements of any team.

02:15:00  alert
02:25:00  flags_off
02:40:00  rollback

The alert fires at 02:15:00, the tools are disabled at 02:25:00, and the alias moves back to the previous bundle at 02:40:00. The tool flag is first because it is the faster action, one setting versus a verified bundle change, and it stops the irreversible effects, so the order matters.

Containment has gaps and costs

Neither action takes effect everywhere at once. The alias cache means a repoint at 02:40:00 reaches a replica only when its cached entry expires. The triage script, which selects requests by the bundle that served them, shows the gap.

selected by time window 02:00:00-02:40:00: 120
selected by served bundle r-57:                 123
  served r-57 after the rollback time:          7
  in window but served r-56 (not affected):     4
  last r-57 request:                            02:41:16

The time window from the release to the rollback contains 120 requests. Selecting by bundle finds 123. Seven requests were served by the bad bundle after the rollback time, and the last arrived at 02:41:16, 76 seconds after the repoint. Four requests in the time window were served by the good bundle and were not affected. A time window is a convenient approximation, and in this run it misses requests that the bad release affected and includes requests it did not. Each log record must therefore carry the identifier of the bundle that served it, which is what makes an accurate selection possible. Recording the served bundle in the response metadata is also how an operator verifies that a rollback has taken effect everywhere. A command that reported success is not evidence that all replicas stopped serving the old version.

Containment also costs something. With the tool flags off from 02:25:00 to the end of the simulated day, 269 tool calls were stopped. Only two of them were the wrong refunds the flag was meant to prevent. Of the rest, 254 came from the restored bundle behaving correctly under policy and 13 were in-policy requests still served by the bad bundle, and all of them now wait for a person. A tool flag that is never turned back on turns an incident into an outage of the automated path, and the on-call engineer should know what it costs per hour to leave a mitigation in place. That is why the recovery section below defines when and how to turn things back on.

Provider failures

An external model provider can fail in ways that differ from a database failing. It can go down, return elevated errors, rate-limit a tenant, respond slowly, silently degrade quality, retire a model version, or change behavior within a version. The first three show up in operational signals. The last three are quiet and look like the incident above.

The standard mitigation is failover to another provider, model, or deployment option, as covered in the gateway article. During an incident, failover is itself a change, with at least three consequences.

• Quality differs. The fallback model has not been through the evaluation gate for this prompt and tool set. A prompt tuned for one model can behave differently on another, and tool-calling reliability can differ too. The safe default is to fail over in a degraded mode: read-only, or with write tools disabled until the fallback passes the regression suite.
• Data handling differs. The terms under which a provider processes and retains data are part of the contract with the data’s owner. Provider documentation shows the variation. Anthropic’s data retention page describes zero data retention as available for eligible features and not others, lists batch processing and the Files API as not eligible, and notes that on Amazon Bedrock and Google Cloud the cloud provider, not Anthropic, is the data processor, with its own controls. Moving a request from one provider or feature to another can therefore move restricted data under different terms. The failover policy must know which data classes may go where.
• Retries amplify. A provider outage that every client retries aggressively can overload the provider’s recovery and the platform’s own queues, and each retry costs money. Backoff, retry budgets, and a circuit breaker belong in the gateway.

The following script encodes a failover decision with those constraints. It is a small policy function, not a gateway. The primary provider is down. The secondary has standard retention and has not passed the write-tool gate. Data classes require zero retention for restricted data.

# Provider failover that respects data-handling terms and write-tool eligibility.
PROVIDERS = {
    "primary":   {"retention": "zero", "write_tools_gated": True,  "up": False},
    "secondary": {"retention": "standard", "write_tools_gated": False, "up": True},
}
ALLOWED_RETENTION = {"public": {"zero", "standard"}, "internal": {"zero", "standard"}, "restricted": {"zero"}}

def route(data_class, needs_write_tools, providers):
    for name in ("primary", "secondary"):
        p = providers[name]
        if not p["up"]: continue
        if p["retention"] not in ALLOWED_RETENTION[data_class]: continue
        if needs_write_tools and not p["write_tools_gated"]: return name, "read-only"   # no regression gate passed
        return name, "full"
    return None, "queue_for_human"

for dc, wt in [("public", False), ("internal", False), ("internal", True), ("restricted", False)]:
    target, mode = route(dc, wt, PROVIDERS)
    print(f"{dc:10} write_tools={str(wt):5} -> provider={target}, mode={mode}")

public     write_tools=False -> provider=secondary, mode=full
internal   write_tools=False -> provider=secondary, mode=full
internal   write_tools=True  -> provider=secondary, mode=read-only
restricted write_tools=False -> provider=None, mode=queue_for_human

Public and internal data fail over to the secondary at full function when the request does not need write tools. An internal request that needs write tools fails over in read-only mode. Restricted data finds no eligible provider and is queued for a human, because its allowed retention terms exclude the secondary. A failover that did not consult the data class would have sent the restricted request to the secondary and exposed the platform to a data-handling incident on top of the outage. The configuration the function reads, the allowed terms and the write gate, is decided before the incident, since the middle of an incident is a poor time to decide who may see what.

Artifact rollback

What a release is made of

A release of an AI feature is a bundle of independently versioned artifacts, the kind the artifact versioning article treats for models, data, prompts, and evaluation artifacts: the model version, the prompt and its template, tool definitions and policies, the retrieval index and the embedding model that built it, configuration such as thresholds and routing rules, and the evaluation set that gated it. A failure can come from any one of them or from their interaction. That article treats promotion and rollback as operations on a combination of artifacts, recorded in a release manifest, and this article calls the combination a release bundle. Rolling back the wrong artifact leaves the fault in place.

The simulation makes this visible. The fictional release r-57 changed three artifacts at once: model m-v12, prompt p-41 (which in this story dropped the sentence requiring human approval above the limit), and index i-1120 (which in this story ranks an outdated policy chunk first). The rules that decide the assistant’s behavior depend on the prompt and the index and ignore the model version, which is the situation where the model is not the faulty artifact. A regression suite of 600 cases, half policy questions and half refund requests, was run against every combination of the old and new artifact.

combination (model, prompt, index)          wrong_days  over_limit_refunds  needless_escalations
m-v11  p-40   i-1030                                 18                   0                    5  <- r-56
m-v11  p-40   i-1120                                540                   0                    5
m-v11  p-41   i-1030                                 16                  90                    4
m-v11  p-41   i-1120                                534                  90                    4
m-v12  p-40   i-1030                                 18                   0                    5
m-v12  p-40   i-1120                                540                   0                    5
m-v12  p-41   i-1030                                 16                  90                    4
m-v12  p-41   i-1120                                534                  90                    4  <- r-57

The row for the fictional release shows 534 answers with the wrong return window and 90 over-limit refunds. The pairs of rows that differ only by the model (m-v11 and m-v12) are identical, so the model is not the cause. Prompt p-41 is the cause of the 90 over-limit refunds, and the index i-1120 is the cause of the wrong return windows, with 534 and 540 against 16 and 18 where the old index is used. A team that suspected the new model first and rolled back only the model alias would have kept both faults, as the row with m-v11, p-41 and i-1120 shows. Rolling back the whole bundle works without knowing which artifact is at fault, which is why the bundle is the unit of rollback during the incident and the combination test is the unit of diagnosis after it.

What rollback needs to be possible

Rollback is not a free operation, and the conditions for it are set before the release. The glossary entry on rollback states the central caution: an older version is useful only if it can still work with current data, interfaces, and access requirements.

• The old artifacts still exist. Retention rules that delete the previous index or prompt to save storage defeat rollback. The previous bundle stays deployable for as long as the new one is on probation.
• Compatibility holds. A new embedding model produces vectors that an old query encoder cannot search. A new tool schema may not match the old prompt. State written by the new version, such as conversation memory or cached summaries, may be unreadable or misleading to the old one. Compatibility between adjacent bundles is something the release process tests, and each release records what it changed in the data layer.
• The switch is fast and observable. An alias or a pointer in configuration, with the served bundle recorded on every response.
• The caches know. Gateway alias caches, prompt caches, semantic caches, and retrieval caches may hold results produced by the bad version. After a rollback, entries created by the bad bundle should be invalidated or at least be keyed by bundle so they cannot be served. The cost article’s semantic cache warning applies directly: a stored wrong answer keeps being served after the model that produced it is gone.

What can be undone and what cannot

This is the center of the article. Each effect of the bad release falls on one side of a line, and the right response is different on each side.

Effect	Reversible by the platform?	Containment	Recovery or compensation
Model alias, prompt, index version	Yes	Repoint to the previous bundle	Verify the served version; keep the old bundle available
Tool enablement, routing, quotas	Yes	Flag or config change	Re-enable in stages after verification
Cached answers from the bad bundle	Yes	Invalidate by bundle	Confirm no entries remain
Draft records, tickets, internal annotations	Mostly	Disable the writing tool	Revert or relabel the records
Chat answers a user already read	No	Stop the cause	Corrective notice where the user is reachable
Emails and messages already sent	No	Disable the send tool	Corrective message; log which recipients received what
Payments and refunds already issued	No	Disable the payment tool	Business decision per item: accept, recover through the payment process, or write off
Actions in a third party’s system	Depends on the system’s API	Disable the integration	Use the third party’s reversal process if one exists, and record the outcome
Deleted or overwritten data	Only with backups or versioning	Disable the tool	Restore from backup; check consistency afterward

The table drives design as well as response. Before an incident, classify every tool by the reversibility of its effects. Reversible tools can run with fewer approvals because a mistake costs a revert. Irreversible tools need an independent gate, such as the approval and idempotency controls of the agent runtime article, or a limit enforced in the tool. The fictional incident exposes a design fault of exactly this kind. The refund limit existed only as a sentence in a prompt, so when a prompt edit removed the sentence, the limit disappeared. The structural fix puts the limit in the tool, where no prompt change can reach it. A minimal version, with its output:

# Structural fix: the refund limit lives in the tool, not only in the prompt.
class ApprovalRequired(Exception): pass

def issue_refund(order_id, amount, in_window, approved_by=None, limit=100):
    if (amount > limit or not in_window) and approved_by is None:
        raise ApprovalRequired(f"refund {amount} on {order_id}: needs a human approval")
    return {"order": order_id, "amount": amount, "approved_by": approved_by or "policy"}

print(issue_refund("o-1", 80, True))
try:
    issue_refund("o-2", 220, True)
except ApprovalRequired as e:
    print("blocked:", e)
print(issue_refund("o-2", 220, True, approved_by="agent-17"))

{'order': 'o-1', 'amount': 80, 'approved_by': 'policy'}
blocked: refund 220 on o-2: needs a human approval
{'order': 'o-2', 'amount': 220, 'approved_by': 'agent-17'}

The tool raises when a refund exceeds the limit or falls outside the return window without an approver, and it proceeds with an approver recorded. The prompt can still contain the rule, which helps the model avoid wasted attempts, but the control no longer depends on it. This is the same idea as the invariant row in the signals table: a check at the boundary of the irreversible effect detects the violation at the moment it would happen, and in its enforcing form prevents it.

Finding what already happened

After containment, the irreversible effects must be located. The triage script reads the structured request log and applies the incident’s invariants to each record. It selects by the bundle that served the request, and it separates effects by their remedy.

by_window = [r for r in log if release <= r["ts"] < rollback]
by_bundle = [r for r in log if r["bundle"] == "r-57"]
print(f"selected by time window {fmt(release)}-{fmt(rollback)}: {len(by_window)}")
print(f"selected by served bundle r-57:                 {len(by_bundle)}")
print(f"  served r-57 after the rollback time:          {sum(r['ts'] >= rollback for r in by_bundle)}")
print(f"  in window but served r-56 (not affected):     {sum(r['bundle'] != 'r-57' for r in by_window)}")
print(f"  last r-57 request:                            {fmt(max(r['ts'] for r in by_bundle))}")

wrong_refund, wrong_email, wrong_answer, ok_refund = [], [], [], []
for r in by_bundle:
    if r["kind"] == "refund" and executed(r, "issue_refund"):
        if policy_decision(r["amount"], r["in_window"]) == "escalate": wrong_refund.append(r)
        else: ok_refund.append(r)
        if executed(r, "send_email") and r["stated_days"] != RETURN_DAYS: wrong_email.append(r)
    if r["kind"] == "question" and r["stated_days"] != RETURN_DAYS: wrong_answer.append(r)

The output on the simulated day:

A. refunds executed that policy required a human to approve: 8, total $1470
   of which over the amount limit: 6, outside the return window: 2
B. confirmation emails stating the wrong return window: 22
C. chat answers stating the wrong return window (no email): 58
D. in-policy refunds executed under r-57 (sample, no action): 18
E. tool calls stopped by the flag after 02:25:00: 269
   would have been wrong refunds (r-57, policy said escalate): 2, $280
   collateral: in-policy refunds from r-56 now waiting for a human: 254

Eight refunds worth 1,470 dollars were executed that policy required a human to approve: six over the amount limit and two outside the return window. Twenty-two confirmation emails stated the wrong return window. Fifty-eight chat answers did too, with no email attached. Eighteen refunds executed under the bad bundle were in policy and need only sampling. The flag stopped two further wrong refunds worth 280 dollars that arrived after 02:25:00 but before the alias propagated, which is the benefit of acting with the flag before the rollback.

Each category has its own remedy, so they are separate worklists. The eight refunds go to a finance review that decides per item. The emails need a correction to the recipients, who are identifiable from the effect records. The chat answers matter if the user can be reached or if the answer led to a decision. The last step splits the lists by tenant, since each tenant’s contact and contract determine who is notified.

worklist by tenant (A refunds, B emails, C answers)
  acme     A=  1 B=  8 C= 18
  birch    A=  4 B=  9 C= 21
  cobalt   A=  3 B=  5 C= 19

This analysis depends on what the platform logged during normal operation. The log needs the request identifier, the served bundle, each tool call with arguments and status, and an effect identifier that ties the platform’s record to the external system’s record. The article on observing AI systems without exposing sensitive data covers keeping such records free of content that should not be stored. Without these fields, the team must reconstruct impact from external systems and customer reports, which is slower and misses cases. An incident also changes how long the logs must be retained: the records that identify affected requests have to outlive the normal retention window until the incident is closed.

Recovery actions that re-run work must not repeat the original effects. If the team replays failed or blocked requests after the fix, each tool call needs an idempotency key so that a second execution recognizes the first. The glossary entry on idempotency covers why the key defines what counts as the same operation. The platform’s effect identifiers serve that purpose here.

Recovery verification

Containment makes the incident stop growing. Recovery is a separate claim: that the system is back to a known-good state and that re-enabling the disabled capabilities is safe. Verification has several parts, and they check different things.

1. The served version is the intended one. Check the bundle identifier recorded on live responses across all replicas, and wait for the cache interval to pass. In the simulation the last bad request came 76 seconds after the repoint.
2. Behavioral signals return to baseline. The metrics that detected the incident, such as refund dollars per window and the audit failure rate, should return to the range seen before the release. A metric that stays elevated after the rollback indicates a second cause or a missed cache.
3. The regression suite passes. Run the evaluation set against the configuration that will serve traffic. Split its results into hard invariants that must be exactly zero, such as an over-limit refund executed without approval, and quality metrics compared with the previous baseline, such as the wrong-return-window rate.
4. The incident is added to the suite. The inputs that triggered the failure become permanent regression cases. A release that reintroduces the fault fails the gate before it ships.
5. Re-enable in stages. Turn capabilities back on in order of reversibility: read-only first, then reversible tools, then irreversible tools under tighter limits and a fraction of traffic, then full function. Hold each stage until its evidence is in.

The fix for the fictional incident is a new bundle, r-58, with a prompt p-42 that restores the approval rule and an index i-1130 rebuilt from the correct policy source, still on model m-v12. It goes through the same regression suite.

candidate fix r-58 (model m-v12, prompt p-42, index i-1130)
regression suite of 600: wrong_days=18 over_limit_refunds=0 needless_escalations=5

There are no over-limit refunds and 18 wrong return windows out of 600, which is the same rate as the old bundle, 3 percent, so the quality metric meets its baseline. The five needless escalations also match the baseline: these are cases where the assistant escalated a refund that policy would have allowed, which costs a human’s time and no money.

A passing suite is evidence about the cases in the suite. For an irreversible capability the other question is how much live evidence is enough before restoring it. If a canary of \(n\) refund requests that needed approval produces no violation, an exact one-sided 95 percent upper bound on the true violation rate is \(1 – 0.05^{1/n}\), which is close to \(3/n\) for large \(n\). Treat the bound as a statement about independent requests from a similar distribution, not a guarantee. It converts a vague “looks fine” into a number the team can compare with a risk tolerance.

upper 95% bound on the violation rate after n clean requests: 1 - 0.05**(1/n)
  n=   50  <= 0.0582
  n=  200  <= 0.0149
  n= 1000  <= 0.0030
  n= 5000  <= 0.0006

After 50 clean requests the rate could still be almost 6 percent. After 1,000 it is at most 0.3 percent, and after 5,000 at most 0.06 percent. A team that decides refunds may resume at a violation rate below 0.5 percent would hold the canary until about 600 qualifying requests and until the structural guard from the previous section is in place, since the guard is what turns a probabilistic claim into a hard limit. The numbers show why staged re-enabling takes time and why low-traffic features need a longer canary than high-traffic ones.

The postmortem

A blameless postmortem is a written review of an incident that focuses on how the system and its processes allowed the failure, not on who made the change. Google’s site reliability engineering book describes the premise that people involved acted with good intentions and that systems and processes are what can be fixed. It lists the contents of a good review: the incident and its impact, the actions taken, the root causes, preventive measures, and key data. NIST’s Playbook emphasizes the same records from a governance angle: keep a database of reported errors, incidents, and responses, and document modifications and deployment details so that improvements can be traced.

For an AI incident, several questions go beyond the usual ones, and the simulation’s story answers each of them.

• Why did no operational alert fire? Because the failure was behavioral. Action: add outcome and invariant monitors for the behavior that failed.
• Why was the limit expressed only in a prompt? Because a policy written as natural language is easy to ship and easy to lose. Action: enforce limits in the tool.
• Why did the evaluation gate pass the bundle? Either it had no cases for approval limits and return windows, or its thresholds allowed the regression. Action: add the incident’s cases and a hard-zero invariant.
• Why did three artifacts ship together without being tested separately? Bundling makes rollback simple and diagnosis hard. Action: record which artifact changed what, and run the combination test automatically on failure.
• How long between release, detection, containment, and verified recovery, and which step was slow? In the simulation the alert came 15 minutes after the release and containment 25 minutes after that. Action: shorten the slowest step, for example with pre-approved flag changes that do not wait for review.

Action items need owners and due dates, and the review should list the effects that could not be undone and who is handling each. A postmortem that lists only fixes to the system and omits the affected customers has skipped the part of the incident that was irreversible.

What these experiments do not show

• The assistant is a rule set. A real model’s failures are stochastic and rarely map to one artifact. Diagnosing a real regression needs repeated runs per combination and a statistical comparison, not a single pass.
• The timings are invented. Detection in 15 minutes depends on a large effect and a well-chosen metric, and the acknowledgment and rollback delays are assumptions. Real response depends on paging, access, and authority to act.
• The failover function is a toy. A real gateway deals with partial failures, streaming responses, and in-flight requests.
• Legal and contractual duties are outside the scope. Incidents involving personal data, payments, or regulated domains can carry notification obligations with deadlines. Those depend on jurisdiction and contract and need the organization’s legal and security functions, who should be named in the runbook before an incident.
• Recovery cannot undo everything. The effect table is the honest summary: the platform controls reversible effects and limits the growth of the others.

The next article, Case: Enterprise Knowledge Search with Permissions and Freshness, applies these controls to a fictional enterprise search assistant. There the irreversible effect is a document shown to someone not entitled to see it, the artifacts include an index whose contents change daily, and the questions of containment, rollback, and verified recovery above are asked about a system that has to stay both permission-correct and fresh.

References

• Google SRE Book, Managing Incidents
• Google SRE Workbook, Incident Response
• Google SRE Book, Postmortem Culture: Learning from Failure
• NIST AI RMF Playbook, Manage
• Anthropic documentation, API and data retention