Case: An Approved Agent for Transactional Work

Consider a support representative who pastes a ticket into an assistant: a customer received a cracked item, wants a refund of \$120, and would like a replacement shipped. An agent can read the order, issue the refund, and request the reshipment. The risks of letting it do so are not the ones that make demonstrations nervous. The model rarely invents a customer. The failures that cost money are quieter: the refund call times out after the ledger committed it and a retry pays twice; the process restarts between the refund and the reshipment and starts the ticket over; a lead approves a \$120 refund and the plan is edited to \$450 before it runs; four refunds of \$49 each stay under an approval threshold of \$50; a configuration change quietly removes the threshold; and weeks later nobody can say which refunds the agent made or whether the ledger agrees.

This article builds the system that prevents, detects, and contains those failures, and runs it. It continues in the same fictional company as Case: Enterprise Knowledge Search with Permissions and Freshness. Quillfeather Logistics, its representatives, team leads, orders, ledger, and order system are all invented for this article. No real organization or incident is described, and the measured behavior comes from a local simulation: a rule-based planner stands in for the model, the ledger and order system are small programs, time is simulated, and every number is the output of running that code. The planner is deliberately obedient. It converts ticket text into tool calls without judgment, so the controls are tested against the worst case, a model that does whatever the text says.

The article assumes you have read Agent Runtime Isolation, Tools, and Execution Policies, which introduced the runtime’s policy engine, approvals bound to one call, idempotent execution, run state, and a hash-chained audit, and AI Platform Incident Response and Recovery, whose containment ladder and distinction between reversible and irreversible effects are used in the incident drill. AI Platform Security and Tenant Isolation supplies the habit of testing a control by removing it. The method of defining invariants and checking them against an oracle is the one used in Case: Enterprise Knowledge Search with Permissions and Freshness. This article does not repeat those explanations. It adds what integration requires: one system in which authorization, approval, durable state, reconciliation, and containment depend on one another.

After reading, you should be able to specify which parts of a transactional workflow need authorization, approval, and exactly-once handling and why; explain how an approval stays tied to the exact action; reason about every crash point between a decision and its recorded effect; reconcile agent records with the systems of record and classify the differences; and contain an incident by switching tools off at execution time, selecting the affected work from records, and compensating what cannot be undone.

The code ran with Python 3.12.14 and SQLite 3.53.4 on macOS 26.7.1, using only the standard library. Runs are deterministic.

The work, the rules, and the failures to prevent

Tickets arrive from customer-support representatives. The agent has three tools. lookup_order reads an order. issue_refund posts a refund to the payments ledger. reship_order asks the order system to create a new shipment. Each tool acts for the representative who owns the ticket, within what that representative may do; the agent has no authority of its own beyond that. Representatives belong to a region. Team leads approve what exceeds the representatives’ authority. The rules the company chose are in the table.

Rule	Value in this system
Refunds up to the auto limit	\$50.00 per order, counting refunds already made on that order. Executed without approval
Refunds above the auto limit	Up to \$500.00. Require approval from a lead in the same region whose own limit covers the amount, and who is not the requester
Refunds above the maximum	More than \$500.00 is denied; a person uses the normal refund tool
Refund cannot exceed what was paid	Counting every refund already recorded for the order
Reshipping	Allowed for shipped orders. Approval is required when the order was paid more than \$300.00
Caps	\$1,000.00 per representative per rolling day; \$3,000.00 per hour across the agent
Approval lifetime	15 minutes, single use
Delegation	A representative’s token ceiling for refunds is \$500.00; a representative with no refund scope cannot refund through the agent

These rules become five invariants, each of which a test or a query can falsify.

Invariant	Statement	Where it is checked
Authority	Every executed effect traces to a user’s delegation and stays within it, even if the runtime’s policy is wrong	Runtime policy plus the downstream system’s own check of the credential
Approval	A refund above the auto limit executes only with a used approval that names exactly that call	Approval binding; a standing query over records
Exactly-once effect	Retries, restarts, and repeated proposals produce one effect per logical action, or an explicit unknown	Idempotency keys, write-ahead records, lookup
Agreement	The agent’s records and the systems of record agree, or each difference is classified and owned	A reconciliation job that does not trust the agent’s status
Containment	Any tool can be stopped at the moment of execution, the affected work can be selected, and the effects can be compensated or accepted by a person	Switches read at execution, credential epochs, records

Design decisions and what each gives up

Decision	Chosen	Alternative	What the choice gives up
Credentials	A token per call: subject is the representative, actor is the agent, audience is one system, scope is one operation, ceiling is the representative’s delegation, 60 seconds, bound to a revocation epoch	One shared service credential for the agent	A credential-minting step on every call, and an epoch to manage. In return, the agent can never exceed the person it acts for
Where limits live	In the runtime policy, and again as the token ceiling the ledger enforces	Only in the runtime	Two places hold a number and can drift. In return, a wrong policy is bounded by the delegation
Approval	A record bound to a hash of the workflow, tool, and normalized arguments; summary built by the runtime; requester cannot approve; lead of the same region within their limit; single use; 15 minutes	Approve “the agent’s run” as a whole	More approvals when plans change. In return, the approver sees and authorizes what executes
Threshold basis	Cumulative refunds on the order	Per call	More approvals for legitimate multi-part refunds. In return, splitting does not evade the threshold
Run state	A step table in a database, with a write-ahead mark before each side effect	In-memory loop; or a full workflow engine	A small engine to maintain; no timers, versioning, or visibility tooling that an engine provides. See the glossary entry on durable execution
Retries	Stable key from workflow, tool, and normalized arguments for the idempotent ledger; at most once plus lookup for the order system	Blind retry everywhere	Some steps end in an explicit unknown instead of finishing
Agreement	An independent reconciliation over steps, ledger entries, and shipments	Trust step status	A job to run and own
Stopping	A per-tool switch read at execution time, plus a credential epoch	A feature flag checked when the ticket arrives	A database read per step

Two of these deserve a comment. The duplicated limit is intentional redundancy and also the main way a configuration can go wrong, which the incident drill exercises. And the runtime is not a workflow engine: it implements the handful of properties the case needs (persisted steps, resumption, keys, waiting for a person) so they can be inspected, and a production system would more likely adopt an engine that provides them and keep the policy, approval, and reconciliation layers above it. The idea of a saga applies as soon as steps must be undone: here, the compensating step for a refund is a ledger reversal that needs a person’s approval.

Implementation

State: workflows, steps, approvals, switches

A workflow row records the ticket, the user, a status, and the policy version in force when it started. A step row holds the tool, the normalized arguments, a status, the idempotency key, the approval it is waiting on, the result, and a baseline observation taken before the side effect. Approvals are separate rows. Switches hold one flag per tool. A decision log records every policy outcome with its rule and the policy version.

CREATE TABLE workflow(id TEXT PRIMARY KEY, ticket TEXT, user TEXT, status TEXT, policy TEXT, created REAL, updated REAL);
CREATE TABLE step(wf TEXT, n INT, tool TEXT, args TEXT, status TEXT, idem_key TEXT, approval TEXT, result TEXT, baseline TEXT,
  attempts INT DEFAULT 0, note TEXT, t_done REAL, PRIMARY KEY(wf, n));
CREATE TABLE approval(id TEXT PRIMARY KEY, wf TEXT, n INT, call_hash TEXT, summary TEXT, requester TEXT, approver TEXT, status TEXT, expires REAL);
CREATE TABLE switch(tool TEXT PRIMARY KEY, enabled INT);
CREATE TABLE decision_log(t REAL, wf TEXT, n INT, tool TEXT, decision TEXT, rule TEXT, policy TEXT);

Step statuses are pending, held (waiting for an approval or a switch), started (intent recorded, effect possibly happened), unknown, done, denied, suppressed (a duplicate proposal), and compensated. A workflow is running, waiting_approval, paused, needs_reconciliation, done, or failed.

Credentials: authority carried by the call

The credential module follows the delegation semantics of RFC 8693: the token names the person whose authority is being used and, separately, the agent acting. It adds two fields the downstream system checks on its own, a ceiling for the amount and the revocation epoch. Raising the epoch invalidates every outstanding token at once, which is the credential half of the kill switch.

class Authority:
    def __init__(self, clock): self.clock, self.epoch = clock, 1
    def mint(self, *, sub, act, aud, scope, max_cents, ttl=60):
        body = json.dumps(dict(sub=sub, act=act, aud=aud, scope=scope, max=max_cents, exp=self.clock() + ttl, epoch=self.epoch), sort_keys=True)
        return base64.b64encode(body.encode()).decode() + "." + hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()
    def verify(self, token, *, aud, scope, amount=None):
        raw, sig = token.rsplit(".", 1); body = base64.b64decode(raw)
        if not hmac.compare_digest(sig, hmac.new(SECRET, body, hashlib.sha256).hexdigest()): raise AuthError("bad signature")
        c = json.loads(body)
        if c["exp"] <= self.clock(): raise AuthError("token expired")
        if c["epoch"] != self.epoch: raise AuthError("credential epoch revoked")
        if c["aud"] != aud: raise AuthError(f"token is for {c['aud']}")
        if c["scope"] != scope: raise AuthError("scope")
        if amount is not None and amount > c["max"]: raise AuthError(f"amount {amount} above token ceiling {c['max']}")
        return c

Policy: ordered rules, default deny, named outcomes

The policy function takes the workflow’s user, the tool, and normalized arguments, and returns a decision with the rule that produced it. It reads the agent’s own records to apply the cumulative and cap rules, and it counts steps that are done, started, or unknown as spent, so an in-flight step is not forgotten. The model contributes only the arguments listed in the tool’s schema; anything else, including a text claim of approval, is dropped during normalization.

def decide(self, wf, user, tool, nargs):
    d = DELEGATIONS.get(user); P = self.policy; t = self.clock()
    if d is None: return "DENY", "R0-unknown-user"
    if TOOLS[tool]["scope"] not in d["scopes"]: return "DENY", "R1-scope-not-delegated"
    o = self.orders.orders.get(nargs["order"])
    if o is None or o["region"] != d["region"]: return "DENY", "R2-order-outside-region"
    if tool == "issue_refund":
        amt = nargs["amount"]
        if amt <= 0 or amt > P["refund_max"]: return "DENY", "R3-refund-over-ceiling"
        q = "SELECT COALESCE(SUM(CAST(json_extract(args,'$.amount') AS INT)),0) FROM step JOIN workflow ON workflow.id=step.wf WHERE tool='issue_refund' AND step.status IN ('done','started','unknown')"
        prior = self.db.execute(q + " AND json_extract(args,'$.order')=?", (nargs["order"],)).fetchone()[0]
        if prior + amt > o["paid"]: return "DENY", "R4-exceeds-amount-paid"
        day = self.db.execute(q + " AND workflow.user=? AND workflow.created>?", (user, t - P["window"])).fetchone()[0]
        if day + amt > P["rep_daily"]: return "DENY", "R5-rep-daily-cap"
        hour = self.db.execute(q + " AND workflow.created>?", (t - 3600,)).fetchone()[0]
        if hour + amt > P["agent_hourly"]: return "DENY", "R6-agent-hourly-cap"
        basis = (prior + amt) if self.flags.cumulative else amt
        if basis > P["refund_auto"]: return "REQUIRE_APPROVAL", "R7-refund-needs-approval"
    if tool == "reship_order":
        if o["status"] != "shipped": return "DENY", "R8-order-not-shipped"
        if o["paid"] > P["reship_approval_value"]: return "REQUIRE_APPROVAL", "R9-reship-needs-approval"
    return "ALLOW", "default-allow-after-checks"

Approval: a record bound to one call

The approval summary is built from the normalized arguments and the order record, never from model text. Approving checks the approver’s role, region, and limit, and refuses the requester. Consuming an approval at execution checks status, expiry, and that the hash of the workflow, tool, and arguments is the one that was approved. An amended step therefore cannot ride on an earlier approval.

def approve(self, aid, approver):
    a = self.db.execute("SELECT wf,n,requester,status,expires FROM approval WHERE id=?", (aid,)).fetchone()
    who = DELEGATIONS.get(approver)
    if a is None or a[3] != "pending": return "not-pending"
    if approver == a[2]: return "refused: requester cannot approve"
    if who is None or who["role"] != "lead": return "refused: approver is not a lead"
    if a[4] <= self.clock(): self.db.execute("UPDATE approval SET status='expired' WHERE id=?", (aid,)); return "refused: expired"
    n = self.db.execute("SELECT tool,args FROM step WHERE wf=? AND n=?", (a[0], a[1])).fetchone(); args = json.loads(n[1])
    reg = DELEGATIONS[self.db.execute("SELECT user FROM workflow WHERE id=?", (a[0],)).fetchone()[0]]["region"]
    if who["region"] != reg: return "refused: approver outside region"
    if n[0] == "issue_refund" and args["amount"] > who["approve_limit"]: return "refused: above approver limit"
    self.db.execute("UPDATE approval SET status='approved', approver=? WHERE id=?", (approver, aid)); return "approved"

def _consume(self, aid, wf, tool, nargs):
    a = self.db.execute("SELECT call_hash,status,expires FROM approval WHERE id=? AND wf=?", (aid, wf)).fetchone()
    if not a or a[1] != "approved": return "no approved request"
    if a[2] <= self.clock(): return "approval expired"
    if self.flags.bind_approval and a[0] != self.call_hash(wf, tool, nargs): return "approval was for a different call"
    self.db.execute("UPDATE approval SET status='used' WHERE id=?", (aid,)); return None

Executing a step

The step function is where the pieces meet. It checks the switch at execution time, applies policy, obtains or consumes an approval, writes the intent and a baseline observation before the side effect, dispatches, and records the result. The three crash hooks mark the points at which a restart must be safe: before dispatch, after dispatch and before the record, and after the record.

def exec_step(self, wf, user, n, tool, args, status):
    nargs = args
    row = self.db.execute("SELECT approval,idem_key,baseline,attempts FROM step WHERE wf=? AND n=?", (wf, n)).fetchone(); aid, key, baseline, attempts = row
    if self.flags.check_switch_at_execution and not self.enabled(tool):
        self.db.execute("UPDATE step SET status='held', note='tool disabled' WHERE wf=? AND n=?", (wf, n)); return "paused"
    if status == "unknown": return self.resolve_unknown(wf, n, tool, nargs, baseline)
    resumed = status == "started"
    if resumed and not TOOLS[tool]["idempotent"]: return self.resolve_unknown(wf, n, tool, nargs, baseline)
    decision, rule = ("ALLOW", "resume") if resumed else self.decide(wf, user, tool, nargs)
    self.db.execute("INSERT INTO decision_log VALUES(?,?,?,?,?,?,?)", (self.clock(), wf, n, tool, decision, rule, self.policy["version"]))
    if decision == "DENY":
        self.db.execute("UPDATE step SET status='denied', note=? WHERE wf=? AND n=?", (rule, wf, n)); return "failed"
    approved_claim = (not self.flags.strict_schema) and bool(args.get("approved_by"))
    if decision == "REQUIRE_APPROVAL" and not approved_claim:
        problem = self._consume(aid, wf, tool, nargs) if (status == "held" and aid) else "none"
        if problem:
            if problem in ("approval expired", "approval was for a different call"):
                self.db.execute("UPDATE approval SET status='void' WHERE id=?", (aid,))
            if problem != "no approved request": self.request_approval(wf, user, n, tool, nargs)
            return "waiting_approval"
        self.db.execute("UPDATE step SET note='approved' WHERE wf=? AND n=?", (wf, n))
    # write-ahead: record intent and the pre-state before the side effect
    if self.flags.write_ahead and not resumed:
        base = json.dumps(len(self.orders.shipments[nargs["order"]])) if tool == "reship_order" else None
        self.db.execute("UPDATE step SET status='started', baseline=?, attempts=attempts+1 WHERE wf=? AND n=?", (base, wf, n))
    self._crash("before_dispatch")
    try: res = self.dispatch(wf, user, tool, nargs, key, n)
    except AuthError as e:                                   # the downstream system refused: record it, do not retry
        self.db.execute("UPDATE step SET status='denied', note=? WHERE wf=? AND n=?", (f"downstream refused: {e}", wf, n)); return "failed"
    if res == "unknown":
        self.db.execute("UPDATE step SET status='unknown' WHERE wf=? AND n=?", (wf, n)); return "needs_reconciliation"
    self._crash("after_dispatch")
    self.db.execute("UPDATE step SET status='done', result=?, t_done=? WHERE wf=? AND n=?", (json.dumps(res, sort_keys=True), self.clock(), wf, n))
    self._crash("after_record")
    return "ok"

Dispatch mints a fresh token for each attempt. For tools whose downstream honors a key it retries once with the same key. For the order system, which has no key, it makes one attempt and reports an unknown outcome when the response is lost. An unknown outcome is resolved by asking the order system what exists and comparing with the baseline recorded before the call.

def dispatch(self, wf, user, tool, nargs, key, n):
    d = DELEGATIONS[user]; spec = TOOLS[tool]
    ceiling = d["refund_cap"] if self.flags.downstream_ceiling else 10 ** 9
    tok = self.auth.mint(sub=user, act=self.agent, aud=spec["aud"], scope=spec["scope"], max_cents=ceiling)
    tries = 2 if spec["idempotent"] else 1
    for i in range(tries):
        try:
            if tool == "lookup_order": return self.orders.lookup(tok, nargs["order"])
            if tool == "issue_refund": return self.ledger.refund(tok, nargs["order"], nargs["amount"], key if self.flags.idempotency_keys else f"{key}-{wf}-{n}-{i}")
            if tool == "reship_order": return self.orders.reship(tok, nargs["order"])
        except Timeout:
            tok = self.auth.mint(sub=user, act=self.agent, aud=spec["aud"], scope=spec["scope"], max_cents=ceiling)
            continue
    return "unknown"

def resolve_unknown(self, wf, n, tool, nargs, baseline):
    """An outcome that could not be observed: ask the downstream system what exists, comparing with the pre-state."""
    if tool == "reship_order" and baseline is not None:
        now = len(self.orders.shipments[nargs["order"]])
        if now > json.loads(baseline):
            self.db.execute("UPDATE step SET status='done', result=?, note='resolved by lookup: shipment found', t_done=? WHERE wf=? AND n=?", (json.dumps({"shipment": self.orders.shipments[nargs['order']][-1]}), self.clock(), wf, n)); return "ok"
        self.db.execute("UPDATE step SET status='pending', note='resolved by lookup: nothing found, safe to repeat' WHERE wf=? AND n=?", (wf, n)); return "running"
    return "needs_reconciliation"

Authorization and approval, exercised

The first run applies the policy to representative requests. Each outcome names its rule, which is what makes denials explainable to the person who made the request and countable in monitoring.

1. policy decisions (each names its rule)
  user     order     amount  decision          rule                         note
  rep-ana  O-1001    $30.00  ALLOW             default-allow-after-checks   small refund in region
  rep-ana  O-1002   $120.00  REQUIRE_APPROVAL  R7-refund-needs-approval     above the auto limit
  rep-ana  O-1002   $600.00  DENY              R3-refund-over-ceiling       above the maximum
  rep-ana  O-1003    $20.00  DENY              R2-order-outside-region      order outside the rep's region
  rep-cy   O-1003    $20.00  DENY              R1-scope-not-delegated       scope not delegated
  rep-ana  O-1001   $150.00  DENY              R4-exceeds-amount-paid       more than the order was paid

The order outside the representative’s region is denied by R2 before any credential is minted, and a representative who never delegated the refund scope is denied by R1. The \$150 request on an order paid \$120 is denied by R4 because it exceeds what the customer paid, a check that holds against a model that has been talked into any amount.

The second run tries to approve a \$400 refund wrongly. Each refusal comes from a different rule, and no refund is made.

2. approval authority: workflow W1 asks for $400 on O-1004 (US, rep-ben)
  summary shown to the approver: Refund $400.00 on order O-1004 for customer-4; requested by rep-ben; workflow W1
  approve as rep-ben   -> refused: requester cannot approve
  approve as rep-cy    -> refused: approver is not a lead
  approve as lead-fay  -> refused: above approver limit
  approve as lead-dee  -> refused: approver outside region
  approve as lead-eli after 1000 s -> refused: expired
  refunds executed: 0

The representative who requested the refund cannot approve it. A colleague without lead status cannot. A lead whose limit is \$100 cannot approve \$400. A lead from the other region cannot. The correct lead is too late once 1,000 seconds have passed, because the approval lived 15 minutes. The summary the approver saw was generated from the normalized arguments and the order record, so it describes the call that will execute.

The third run shows what binding buys. A lead approves \$120, the plan is then amended to \$450 (for example because the representative added a second damaged item to the ticket), and the run is resumed.

3. the same approved call versus a changed one
  approve -> approved
  after amending to $450, run -> waiting_approval | refunds: 0 | new approval: ('AP-2', 'pending')

The amended call hashes differently, so the earlier approval cannot be consumed. The engine voids the mismatched approval, creates a new one for \$450, and waits. Without the binding, the test matrix below shows that the \$450 refund executes on the \$120 approval.

Durable execution: exactly once, or an honest unknown

A step performs three things that cannot be made atomic: it decides, it changes another system, and it records that it did. A crash can fall between any two of them. The write-ahead record before the side effect and the stable key turn the ambiguous cases into recoverable ones. The next table crashes a workflow at each hook, restarts the engine as a new process over the same database file, resumes, and counts effects. The right-hand column repeats the run with the write-ahead record removed.

step     crash point         designed: effects    no write-ahead: effects
refund   before_dispatch           1 refund(s)                1 refund(s)
refund   after_dispatch            1 refund(s)                1 refund(s)
refund   after_record              1 refund(s)                1 refund(s)
reship   before_dispatch           1 reship(s)                1 reship(s)
reship   after_dispatch            1 reship(s)                2 reship(s)
reship   after_record              1 reship(s)                1 reship(s)

The refund survives every crash point in both columns, because the ledger honors the key: a resumed step dispatches again with the same key and the ledger replays its recorded result. This is the reason refunds are safe even without a write-ahead record, and the reason the second column matters only for the reshipment. The order system has no key, so a crash after the shipment was created and before it was recorded leaves a step that never learned it ran. With no write-ahead record, the resumed engine sees only a pending step and creates a second shipment. With the record, the step is started on resume, the engine knows a non-idempotent effect may have happened, and it compares the order’s shipment count with the baseline it saved before the call. One more shipment means the effect happened and the step is marked done; no change means it can be repeated safely. The crash before dispatch and the crash after the record both end with one shipment in either column.

The lookup is a resolution, not a proof. It assumes nobody else reshipped the same order between the baseline and the check. If that assumption is doubtful, the step stays unknown and the workflow stops at needs_reconciliation for a person, which is also what happens when the downstream offers no way to look. The glossary entry on the idempotency key describes this indeterminate state; the point of the engine is to make it a recorded state that someone owns, not a silent retry or a silent skip.

Waiting for a person is also state. When a workflow needs an approval it persists as waiting_approval with the approval row, and nothing keeps a process alive for it. The test approval arrives after a restart starts a workflow, discards the engine, opens a new one over the same file, approves, and runs; the refund happens once. That is the property that makes it practical to wait hours for a lead.

Reconciliation: do the records agree with the world

Status fields say what the agent believes. reconciliation compares the agent’s records with an independent reference, here the ledger and the order system, and treats each difference as something to classify, not to overwrite. The job reads steps, ledger entries, and shipments, joins refunds by idempotency key, and reports eight classes. The run below seeds one example of each failure, then restarts the engine.

matched                                 4  
agent_done_ledger_missing               1  [('WP', 1, 'O-1003', 1000)]
amount_mismatch                         1  [('W0', 2, 'O-1001', 2000, 200)]
ledger_entry_without_step               1  [('L7', 'O-1008', 5000, 'finance-clerk')]
in_flight_ledger_has_it                 1  [('WC', 1, 'O-1007')]
in_flight_ledger_lacks_it               0  []
refund_over_limit_without_approval      0  []
shipment_count_mismatch                 1  [('O-1004', 1, 0)]

Four refunds match. The five seeded differences are all visible. A refund the agent recorded as done is missing from the ledger, which a storage fault that acknowledged and then lost the write produces. A ledger amount of \$2.00 against an agent amount of \$20.00 comes from an adapter bug. A ledger entry by a finance clerk has no step; it is external, and the job reports it without claiming it is wrong. One workflow is in flight with the ledger already holding its entry, which is a crash between effect and record that has not been resumed yet. And an order has a shipment that no step created.

repair class for each non-empty finding
  agent_done_ledger_missing: re-check the ledger; if truly absent, replay with the same key (safe: idempotent) after a human confirms
  amount_mismatch: freeze both records, finance decides which is right; never overwrite silently
  ledger_entry_without_step: classify: external or manual entry (no action) versus lost agent record (adopt by key)
  in_flight_ledger_has_it: resume the workflow; the ledger replays the recorded result

The repair column is a decision about who may act. Resuming an in-flight workflow is safe, because the key makes it idempotent, and restarting the engine resolves it: after the restart there are five matched refunds and no in-flight steps. Adopting a ledger entry as a lost agent record, replaying a refund the ledger lost, and canceling an extra shipment all change money or a customer’s order, so they go to a person. Where two records disagree on an amount, the job freezes both and does not decide. Reconciliation has the limit that its reference can share the agent’s error. If the ledger and the agent both record the wrong amount, they agree. Comparing with something independent, such as the payment processor’s statement, is a further reconciliation layer.

Tests, and what they establish

As in Case: Enterprise Knowledge Search with Permissions and Freshness, each test must pass on the designed system and fail when its control is removed. A test that passes either way is not testing the control.

test                                          designed   control removed  removed control
lost response, retried                            pass              FAIL  idempotency_keys=False
model repeats the same call                       pass              FAIL  run_dedupe=False, idempotency_keys=False
crash at each point, then resume                  pass              FAIL  write_ahead=False
approval arrives after a restart                  pass               n/a  -
plan amended after approval                       pass              FAIL  bind_approval=False
refunds split to stay under the threshold         pass              FAIL  cumulative=False
approval claimed in tool arguments                pass              FAIL  strict_schema=False
policy bug, token ceiling holds                   pass              FAIL  downstream_ceiling=False
tool switched off mid-run                         pass              FAIL  check_switch_at_execution=False
approver authority rules                          pass               n/a  -

layering: the repeated-call test with one layer removed at a time
  run_dedupe=False only: pass | idempotency_keys=False only: pass

Eight tests were run with a control removed, and each failed as it should. With idempotency keys removed, a lost response is followed by a second refund. With the write-ahead record removed, a crash after the shipment duplicates it. With approval binding removed, the amended \$450 refund runs on the \$120 approval. With the cumulative basis removed, four refunds of \$49 on one order all execute. With strict argument normalization removed, a model-written approved_by field is accepted as an approval. With the downstream ceiling removed, a \$700 refund passes a mistaken policy. With the switch check at execution removed, a refund runs after its tool was switched off, because the workflow had already passed the check at admission. The repeated-call test needed both the run-level duplicate check and the ledger’s key removed before it failed, and the last two lines show why: either layer alone catches the duplicate. Layered defenses cannot be mutation-tested one at a time without hiding behind each other, so the test removes both.

The two tests marked n/a have no switch. The restart test and the approver-authority test pass on the designed system and have not been shown to fail on a broken one. What the suite establishes is that these named sequences are handled and that these controls are load-bearing. It does not establish that no sequence leads to a duplicate or an unauthorized refund. It runs one engine against a ledger that never loses idempotency state, and a real concurrent deployment would add races between two engines picking up the same workflow, which needs a lease or a single-writer rule that the sketch does not implement.

An incident: a policy release removes the approval step

The last drill combines everything. Over three simulated hours, tickets arrive on average every 90 seconds. About 70% request a refund under \$50, 22% between \$51 and \$500, and 8% above \$500. About a third also ask for a reshipment. The drill raises the caps to \$5,000 per representative per day and \$10,000 per hour so that they do not dominate the traffic. Leads check their approval queue every two minutes and approve 60% of what is pending. One hour in, a new policy bundle, p2, is released. Its author expressed the auto limit and the maximum in dollars where the engine reads cents, so the auto limit is \$5,000 and the maximum \$50,000. The policy change gate from Prompt and Model Promotion with Evaluation Gates would have rejected it; the gate here is a table of boundary cases, and the next output shows candidate p2 failing three of them.

4. a policy change gate: the same cases run against a candidate bundle
  p1 (current)                     cases failing: 0 []
  candidate with a units mistake   cases failing: 3 [(51, 'REQUIRE_APPROVAL', 'ALLOW'), (500, 'REQUIRE_APPROVAL', 'ALLOW'), (501, 'DENY', 'ALLOW')]

The bundle was released without running that gate. The drill follows what happens. The delays between steps (ten minutes from alert to containment, fifteen to a corrected policy) are assumptions of the simulation, not measurements.

timeline (minutes after the policy release):
  deploy    +  0.0
  alert     +  5.0
  contain   + 15.0
  fix       + 30.0
  reenable  + 35.0
workflows paused at the time of the fix: 11 | after re-enabling they ended as: {'failed': 1, 'done': 10}
policy   refund steps  executed   above $50, no approval   blocked by token ceiling  denied by policy
p1                 29        27        0 (    $0.00)                          0                 2
p2                 20        17        3 (  $827.00)                          1                 2
p3                 69        59        0 (    $0.00)                          0                 7

reconciliation worklist: 3 refunds executed above the auto limit without an approval, total $827.00
  by rep: {'rep-ana': 2, 'rep-ben': 1} | first at +1.8 min | last at +13.8 min
remediation: ratified 1; reversed 2 in the ledger, $769.00 to recover from customers (outside the system)
reconciliation after remediation: {'agent_done_ledger_missing': 0, 'amount_mismatch': 0, 'ledger_entry_without_step': 0, 'in_flight_ledger_has_it': 0, 'in_flight_ledger_lacks_it': 0, 'refund_over_limit_without_approval': 0, 'shipment_count_mismatch': 0} | matched: 101
since the fix: 25 refunds executed under the auto limit, 34 executed with a used approval; violations: 0
  n=34: 95% upper bound on the violation rate with zero violations = 0.0843
  n=300: 95% upper bound on the violation rate with zero violations = 0.0099
  n=1000: 95% upper bound on the violation rate with zero violations = 0.0030

Detection came from an invariant, not from a symptom. A monitor ran every five minutes and asked the records one question: has any refund above the auto limit executed without a used approval? The first violation happened 1.8 minutes after the release and the monitor saw it at 5.0. Containment at 15 minutes did two things at once. It switched the refund tool off in the switch table, which the engine reads at each step, so workflows already in progress stopped at their next refund instead of finishing it. And it raised the credential epoch, which invalidated tokens already minted. The last violation was at 13.8 minutes, so nothing leaked after containment. Eleven workflows were paused at the time of the fix. After the corrected bundle p3 was released and the tool was re-enabled, those eleven resumed under the corrected policy: ten finished, with approvals where required, and one failed because its \$684.00 refund exceeds the corrected \$500.00 maximum.

The blast radius had two bounds. The p2 workflows executed 17 refunds, of which 3, totaling \$827.00, went above the auto limit without approval; the misconfigured maximum let a request above \$500 reach the ledger, and the ledger refused it against the representative’s \$500 ceiling. That refused request is the second copy of the limit doing its job. The damage that did occur was bounded by the delegation and by the number of tickets in the 14-minute window, not by the unit error.

Scoping used the records, not the clock. The reconciliation worklist selects refunds by their own properties (amount, approval status, rep) and not by the time window of the bad bundle, because a time window misses effects of paused workflows and includes unaffected ones. It listed three refunds, two by one representative and one by the other. Compensation is the part that cannot be automated, and it follows the split in AI Platform Incident Response and Recovery between effects the platform can reverse and effects that left it. A lead reviewed each item. One was ratified, with a retroactive approval recorded as such because the ticket was a damaged-goods case with a replacement shipment. Two were reversed in the ledger, which creates a reversal entry but does not return money: \$769.00 remains to be recovered from the customers by a process outside the system. After remediation, reconciliation shows zero unexplained differences and 101 matched refunds. That statement is about agreement between records, and it does not say the company recovered the money.

Re-enabling was staged by evidence. After the fix there were 25 auto-approved refunds and 34 refunds with a used approval, and no violations. Zero violations in 34 approval-gated refunds bounds the violation rate at 8.4% with 95% confidence by the exact one-sided formula \( 1 – 0.05^{1/n} \), which is close to \( 3/n \). That number is too weak to relax anything, which is the useful reading: the evidence supports the corrected policy as designed, and it would take about 300 clean requests to reach 1% and about 1,000 to reach 0.3%. The structural controls, not the sample, carry the guarantee: the ceiling, the approval binding, and the invariant monitor.

The postmortem questions follow the same pattern as the earlier incident article. The monitor fired because the invariant was written at the effect, not at the configuration. The limit existed in two places and the second one limited the damage. The policy bundle shipped without its boundary cases, so the case table becomes a required check on every bundle. The approval basis was correct and was bypassed by configuration, so a configuration change that raises a limit by orders of magnitude should itself require approval. And an alert for “approvals per hour fell to zero” would have detected the failure even before a violating refund was executed, which suggests a second monitor on process shape in addition to the one on outcomes.

What remains unsolved

• Concurrency. The engine assumes one process works a workflow at a time. Two workers picking up the same workflow need a lease or a single-writer partition, and the failure modes of a stale lease are their own design.
• Downstream semantics. The ledger here honors keys perfectly, rejects mismatched parameters, and remembers every key. Real services expire keys, return errors that hide committed effects, and have rate limits. Each such property changes which steps are safe to retry.
• Lookup is not proof. Resolving an unknown reshipment by counting shipments is correct only when nothing else creates shipments for that order concurrently.
• The planner is a stub. A real model produces ambiguous plans, hallucinated order numbers, and adversarial text. The controls here do not depend on its behavior, but the rate at which it proposes useful, correct actions is a separate measurement and a separate cost.
• Approval quality. A lead who sees many requests may approve without reading. The threshold, the summary, and the queue design determine whether approval is a control or a ritual, and the simulation’s 60% approval rate says nothing about human attention.
• Compensation is a business process. A ledger reversal records an intention. The money, the customer relationship, and any legal duty sit outside the system.

The system keeps authority at the person’s level, makes approvals specific to the action, treats every step as something that may need to be resumed or explained, compares its own records with the world, and can be stopped at the moment of effect. The previous case read documents on behalf of employees and the one before it governed what a coding agent may run. The next case, Case: Deploying AI Development Tools Across Engineering Teams, turns to the organization around those tools: how a company introduces them to many teams, measures whether they help, and decides how far to go.

References

• Stripe API reference: Idempotent requests
• OWASP Gen AI Security Project: LLM06:2025 Excessive Agency
• IETF RFC 8693: OAuth 2.0 Token Exchange