Durable Agent Workflows

Consider a hypothetical order-support agent that resolves a damaged-item claim. The workflow has five steps: create a return label, schedule a carrier pickup, refund 42.00 USD, email the customer, and close the ticket. A worker process runs them in order. At the refund step the worker sends the request, the refund service accepts it and moves the money, and a deployment kills the worker before the response arrives. A replacement process picks up the ticket. It knows the ticket is open. It does not know whether the refund happened.

Every available move has a cost. Starting over creates a second label and may refund twice. Skipping to the email trusts a refund that may never have been issued. Asking the model what to do next may produce a different plan from the one that was half-executed. Marking the ticket failed leaves a customer with a pickup scheduled for an item that was never refunded. None of these problems come from the model’s reasoning. They come from running several side-effecting steps in sequence on machines that crash, over networks that lose responses.

The previous article in this series gives the agent a task state, a plan, budgets, and termination rules. The article before it gives each tool call a typed contract, authorization, and a key so that a repeated write is not repeated. Both stop at the same boundary: they assume the process that holds the state stays alive. Here the process can die at any line. A small durable executor, broken deliberately at every point where a process can die, shows which mechanism prevents which failure. Retrying calls to a model API is covered in Integrating Model APIs and is not repeated here; the subject is multi-step work whose steps change the outside world.

All programs below are Python 3.12 using only the standard library. The downstream systems are in-memory fakes with scripted faults, and the workflow input is synthetic. Nothing calls a real service.

What can go wrong around one effect

A step is a unit of work with an effect outside the process: a refund, a label, an email. An attempt is one try at a step. An agent loop and a fixed pipeline differ in who chooses the next step (a model or code), but once a step is chosen, both face the same question: what happened to this attempt?

From the caller’s side an attempt ends in one of three ways: a response that says it worked, a response that says it was refused, or no usable response. The third covers a timeout, a reset connection, and the death of the caller itself. The service’s side has one more distinction: whether the effect was applied. That gives five cases, which the fake services in this article produce on demand.

Case	What the service did	What the caller sees	Name in the code
success	applied	a success response	none
unavailable	not applied	an error before processing, such as a 503	unavailable
timeout, applied	applied	nothing	timeout_after
timeout, lost	not applied	nothing	lost
rejected	refused for good	a permanent error, such as a business rule violation	reject

The two timeout rows look the same to the caller and mean opposite things for the world. That asymmetry is the source of every duplicate and every stuck workflow discussed below. Partial Failure develops it for services in general; what matters here is how it combines with a process that can die between any two lines.

Take the refund step and mark the places where the worker can die:

Crash point	What the journal says	What the world contains	What a restart can tell
before any record of the step	nothing	nothing	the step has not started
after the intent is recorded, before the call	intent	nothing	an attempt may have begun
after the call returns, before the result is recorded	intent	the refund	an attempt may have begun
after the result is recorded	done, with the result	the refund	the step is finished

The second and third rows have identical journals and different worlds. A restarted worker sees “intent” and cannot tell which row it is in, exactly as a caller that timed out cannot tell which timeout it had. The design requirement is therefore stated once for both situations: a step that has an intent but no result is in an unknown state, and the system must be correct whichever world it is in. Every mechanism in this article is a way of meeting that requirement, and each one is evaluated against it.

Durable steps

A durable step is a step whose progress survives the process. The mechanism is a journal, a table that outlives the worker, with one row per step holding its state, its attempt count, and its result. Three rules make it work.

1. Record the intent before the call. Writing “attempt 1 of issue_refund is starting” before sending the request is what lets a restarted worker know an attempt may have happened. Without it, the second row of the crash table is invisible and looks like the first.
2. Record the result after the call, and reuse it on replay. When the worker restarts, it walks the workflow from the beginning. For each step the journal has marked done, it takes the stored result and does not call the service again. That is how later steps get the label ID that an earlier step produced.
3. Make the step’s identity stable. The journal row and any key sent to a service are derived from the workflow ID and the logical step name, never from a counter, a random value, or the model’s own tool-call ID. A re-asked model call can produce a new tool-call ID for the same step; a stable identity cannot change.

The fake services

The first program defines the downstream systems. Each operation applies its effect to in-memory state, and faults scripts what happens on a given call: unavailable and reject raise before the effect, lost raises without applying it, and timeout_after applies the effect and then raises. Three operations accept an idempotency key and store the result under it, so a repeated call with the same key returns the first result. send_email accepts no key and has only a lookup, because real systems of both kinds exist and the difference matters later.

"""Fake downstream systems with fault injection. Python 3.12, standard library only."""

class Timeout(Exception): pass       # outcome unknown: the request may or may not have been applied
class Unavailable(Exception): pass   # clean failure: the request was not applied
class Rejected(Exception): pass      # permanent refusal: retrying cannot help

class World:
    """faults maps (operation, call number) to 'unavailable', 'reject', 'lost' or 'timeout_after'."""
    def __init__(self, faults=None):
        self.faults = dict(faults or {})
        self.calls = {}
        self.keys = {}                                   # (operation, key) -> stored result
        self.labels, self.pickups, self.refunds = {}, {}, []
        self.emails, self.tickets = [], {}

    def _call(self, op, effect):
        n = self.calls[op] = self.calls.get(op, 0) + 1
        fault = self.faults.get((op, n))
        if fault == "unavailable": raise Unavailable(op)
        if fault == "reject": raise Rejected(op)
        if fault == "lost": raise Timeout(op)            # request never applied
        result = effect()
        if fault == "timeout_after": raise Timeout(op)   # applied, but the response is lost
        return result

    def _keyed(self, op, key, effect):
        def run():
            if key is not None and (op, key) in self.keys:
                return self.keys[(op, key)]              # a repeat returns the recorded result
            result = effect()
            if key is not None: self.keys[(op, key)] = result
            return result
        return self._call(op, run)

    def create_label(self, order, key=None):
        def effect():
            lid = f"lbl_{len(self.labels) + 1}"
            self.labels[lid] = {"order": order, "status": "active"}
            return {"label": lid}
        return self._keyed("create_label", key, effect)

    def void_label(self, label):                         # naturally idempotent: voiding twice is voiding once
        def effect():
            if label in self.labels: self.labels[label]["status"] = "void"
            return {}
        return self._call("void_label", effect)

    def schedule_pickup(self, label, key=None):
        def effect():
            pid = f"pu_{len(self.pickups) + 1}"
            self.pickups[pid] = {"label": label, "status": "scheduled"}
            return {"pickup": pid}
        return self._keyed("schedule_pickup", key, effect)

    def cancel_pickup(self, pickup):
        def effect():
            if pickup in self.pickups: self.pickups[pickup]["status"] = "cancelled"
            return {}
        return self._call("cancel_pickup", effect)

    def issue_refund(self, order, cents, key=None):
        def effect():
            self.refunds.append({"order": order, "cents": cents})
            return {"refund": len(self.refunds)}
        return self._keyed("issue_refund", key, effect)

    def send_email(self, to, ref):                       # no idempotency key; the reference is only a label
        def effect():
            self.emails.append({"to": to, "ref": ref})
            return {}
        return self._call("send_email", effect)

    def find_email(self, ref):                           # a lookup by reference, the only duplicate guard here
        return any(e["ref"] == ref for e in self.emails)

    def close_ticket(self, ticket):
        def effect():
            self.tickets[ticket] = "closed"
            return {}
        return self._call("close_ticket", effect)

The journal and the executor

The executor keeps the journal in SQLite. A wf row holds the workflow status and a lease; a step row holds one step’s state. Three flags switch mechanisms on and off so that the later experiments can measure what each one buys: journal (without it, progress lives in memory and is lost on restart), stable_keys (without it, every attempt sends a fresh key), and lookup (whether an unknown outcome is resolved by searching for the effect). The tick method marks the places where the process can be killed, and crash_at picks which tick fails.

"""A small durable step executor over a SQLite journal. Python 3.12, standard library only."""
import itertools, json, sqlite3
from world import World, Timeout, Unavailable, Rejected

class Crash(Exception): pass     # simulated process death
class Busy(Exception): pass      # another worker holds the lease
class Fenced(Exception): pass    # this worker's lease was taken over

STEPS = [("create_label", "void_label"), ("schedule_pickup", "cancel_pickup"),
         ("issue_refund", None),            # the pivot: no compensation exists after this step
         ("send_email", None), ("close_ticket", None)]
PIVOT, MAX_ATTEMPTS = 2, 3

ACTIONS = {
    "create_label":    lambda w, i, r, k: w.create_label(i["order"], k),
    "schedule_pickup": lambda w, i, r, k: w.schedule_pickup(r["create_label"]["label"], k),
    "issue_refund":    lambda w, i, r, k: w.issue_refund(i["order"], i["cents"], k),
    "send_email":      lambda w, i, r, k: w.send_email(i["customer"], k),
    "close_ticket":    lambda w, i, r, k: w.close_ticket(i["ticket"]),
    "void_label":      lambda w, i, r, k: w.void_label(r["create_label"]["label"]),
    "cancel_pickup":   lambda w, i, r, k: w.cancel_pickup(r["schedule_pickup"]["pickup"]),
}
LOOKUP = {"send_email": lambda w, i, r, k: w.find_email(k)}   # steps whose effect can be found again

def new_journal():
    db = sqlite3.connect(":memory:")
    db.executescript("""
      CREATE TABLE wf(id TEXT PRIMARY KEY, input TEXT, status TEXT, note TEXT, gen INTEGER, lease_until REAL);
      CREATE TABLE step(wf TEXT, name TEXT, state TEXT, attempts INTEGER, result TEXT, PRIMARY KEY (wf, name));""")
    return db

class Executor:
    _nonce = itertools.count(1)
    def __init__(self, db, world, wf, *, journal=True, stable_keys=True, lookup=True,
                 crash_at=None, clock=lambda: 0.0):
        self.db, self.world, self.wf = db, world, wf
        self.journal, self.stable_keys, self.lookup = journal, stable_keys, lookup
        self.crash_at, self.ticks, self.clock, self.gen = crash_at, 0, clock, None
        self.mem_steps, self.mem_status = {}, ["running", ""]   # used when journal=False: lost on restart
        self.log = []

    def tick(self):                                  # a point where the process can die
        self.ticks += 1
        if self.ticks == self.crash_at: raise Crash(f"crash at tick {self.ticks}")

    # ---- journal access (one transaction per write; a stale lease holder is fenced out)
    def _fence(self):
        if self.gen is not None and self.db.execute("SELECT gen FROM wf WHERE id=?", (self.wf,)).fetchone()[0] != self.gen:
            raise Fenced(f"generation {self.gen} is stale")
    def get(self, name):
        if not self.journal: return self.mem_steps.get(name, (None, 0, None))
        row = self.db.execute("SELECT state, attempts, result FROM step WHERE wf=? AND name=?", (self.wf, name)).fetchone()
        return (row[0], row[1], json.loads(row[2]) if row[2] else None) if row else (None, 0, None)
    def put(self, name, state, attempts, result=None):
        if not self.journal: self.mem_steps[name] = (state, attempts, result); return
        with self.db:
            self._fence()
            self.db.execute("INSERT OR REPLACE INTO step VALUES (?,?,?,?,?)",
                            (self.wf, name, state, attempts, json.dumps(result) if result is not None else None))
    def status(self):
        if not self.journal: return self.mem_status[0]
        return self.db.execute("SELECT status FROM wf WHERE id=?", (self.wf,)).fetchone()[0]
    def set_status(self, status, note=""):
        if not self.journal: self.mem_status[:] = [status, note]; return status
        with self.db:
            self._fence()
            self.db.execute("UPDATE wf SET status=?, note=? WHERE id=?", (status, note, self.wf))
        return status
    def results(self):
        names = [n for n, _ in STEPS]
        return {n: self.get(n)[2] for n in names if self.get(n)[0] == "done"}

    def claim(self, ttl=30.0):
        now = self.clock()
        with self.db:
            cur = self.db.execute("UPDATE wf SET gen=gen+1, lease_until=? WHERE id=? AND lease_until<=?",
                                  (now + ttl, self.wf, now))
        if cur.rowcount == 0: raise Busy(self.wf)
        self.gen = self.db.execute("SELECT gen FROM wf WHERE id=?", (self.wf,)).fetchone()[0]

    def key(self, jname):
        base = f"{self.wf}:{jname}"                  # derived from the logical step, so it is stable
        return base if self.stable_keys else f"{base}:{next(Executor._nonce)}"

The central method is execute. It follows the three rules directly. The state done returns the stored result without a call. A missing row gets an intent record first. Each attempt increments and persists the attempt counter before the call, then calls with the step’s key. After the call, the effect exists in the world but not yet in the journal, which is the dangerous window; only then is the result recorded.

    def execute(self, jname, action, inp):
        state, attempts, result = self.get(jname)
        if state == "done": return "done", result    # replay: reuse the recorded result, no call
        if state in ("failed", "unknown"): return state, None
        if state is None:
            self.tick(); self.put(jname, "intent", 0)            # write-ahead intent
        elif self.lookup and action in LOOKUP and LOOKUP[action](self.world, inp, self.results(), self.key(jname)):
            self.put(jname, "done", attempts, {"found": True}); return "done", {"found": True}
        unsure = False
        while attempts < MAX_ATTEMPTS:
            attempts += 1
            self.put(jname, "intent", attempts)                    # the attempt counter survives a crash
            key = self.key(jname)
            self.tick()                                            # before the call
            try:
                res = ACTIONS[action](self.world, inp, self.results(), key)
            except Rejected:
                self.put(jname, "failed", attempts); return "failed", None
            except Unavailable:
                continue
            except Timeout:
                unsure = True
                if self.lookup and action in LOOKUP and LOOKUP[action](self.world, inp, self.results(), key):
                    res = {"found": True}
                else:
                    continue
            self.tick()                                            # the effect exists, the journal does not know
            self.put(jname, "done", attempts, res); self.tick()
            return "done", res
        if unsure:
            self.put(jname, "unknown", attempts); return "unknown", None
        self.put(jname, "failed", attempts); return "failed", None

The run method walks the steps in order. A step that is done is skipped, which is replay. A failed step before or at the pivot starts compensation, a failed step after the pivot escalates, and an unknown step escalates. The compensation branch is explained in a later section; it reuses execute for the undo operations, which is why the compensations get the same journaling and keys as the forward steps.

    def finish(self, status, note=""):
        self.set_status(status, note); return status

    def run(self, inp):
        if self.status() == "running":
            for i, (name, _) in enumerate(STEPS):
                outcome, _res = self.execute(name, name, inp)
                if outcome == "done": continue
                if outcome == "unknown": return self.finish("escalated", f"{name}: outcome unknown")
                if i > PIVOT: return self.finish("escalated", f"{name} failed after the pivot")
                self.set_status("compensating", f"{name} failed"); break
            else:
                return self.finish("completed")
        if self.status() == "compensating":
            note = self.db.execute("SELECT note FROM wf WHERE id=?", (self.wf,)).fetchone()[0] if self.journal else self.mem_status[1]
            stuck = []
            for name, comp in reversed(STEPS[:PIVOT]):
                if self.get(name)[0] != "done": continue
                if self.execute("undo:" + name, comp, inp)[0] != "done": stuck.append(comp)
            if stuck: return self.finish("escalated", note + "; could not run " + ", ".join(stuck))
            return self.finish("compensated", note)
        return self.status()

def start(db, wf, inp):
    with db:
        db.execute("INSERT INTO wf VALUES (?,?,?,?,0,0)", (wf, json.dumps(inp), "running", ""))

Two runs

The next program drives the executor. scenario runs the workflow, and if the simulated process dies, a new Executor is created on the same journal and world and resumes. Run 1 has no faults. Run 2 kills the process at tick 11, the third tick of the refund step, which is the window after the refund call and before its result is recorded.

from engine import *
INP = {"order": "ord_1002", "cents": 4200, "ticket": "T-77", "customer": "cu_ana"}

def show(title, db, w, wf="wf1"):
    print(title)
    steps = db.execute("SELECT name, state, attempts FROM step WHERE wf=? ORDER BY rowid", (wf,)).fetchall()
    print("  journal :", ", ".join(f"{n}={s}({a})" for n, s, a in steps))
    status, note = db.execute("SELECT status, note FROM wf WHERE id=?", (wf,)).fetchone()
    print(f"  status  : {status}" + (f" ({note})" if note else ""))
    print(f"  world   : refunds={len(w.refunds)} emails={len(w.emails)} "
          f"labels={[v['status'] for v in w.labels.values()]} pickups={[v['status'] for v in w.pickups.values()]} "
          f"tickets={w.tickets}")

def scenario(title, faults, crash_at=None):
    db, w = new_journal(), World(faults); start(db, "wf1", INP)
    try:
        Executor(db, w, "wf1", crash_at=crash_at).run(INP)
    except Crash as c:
        print(f"  [{c}]")
        Executor(db, w, "wf1").run(INP)               # a new process resumes from the journal
    show(title, db, w)

scenario("1. no faults", {})
scenario("2. crash after the refund call, before its result is recorded", {}, crash_at=11)

1. no faults
  journal : create_label=done(1), schedule_pickup=done(1), issue_refund=done(1), send_email=done(1), close_ticket=done(1)
  status  : completed
  world   : refunds=1 emails=1 labels=['active'] pickups=['scheduled'] tickets={'T-77': 'closed'}
  [crash at tick 11]
2. crash after the refund call, before its result is recorded
  journal : create_label=done(1), schedule_pickup=done(1), issue_refund=done(2), send_email=done(1), close_ticket=done(1)
  status  : completed
  world   : refunds=1 emails=1 labels=['active'] pickups=['scheduled'] tickets={'T-77': 'closed'}

Run 1 shows the journal after a normal workflow: five steps, one attempt each. Run 2 shows the recovery. The journal has issue_refund=done(2): the first attempt applied the refund and died, and the second attempt, from the new process, sent the same key. The fake service recognized it and returned the stored result instead of refunding again, so the world holds one refund and one email. The attempt counter of 2 is the visible trace that a repeat happened. Nothing in the world is duplicated because the key identifies the logical refund rather than the attempt.

The model’s decision is a step too

In an agent, the choice of which action to take is itself a result that must survive. If the restarted worker re-asks the model and the model answers differently, the half-executed plan and the new decision disagree. The following program uses a deterministic stand-in whose answers alternate between calls, which imitates the run-to-run variation of a real model.

"""A model call is a step too: record its output, or a replay can decide differently. Standard library only."""
import itertools
_calls = itertools.count(1)
def stub_model(ticket):                      # deterministic stand-in for a model whose answers vary between calls
    return {"action": "refund", "cents": 4200} if next(_calls) % 2 else {"action": "store_credit", "cents": 4200}

def run_to_crash_and_resume(journal_decision):
    journal = {}                             # survives the crash
    def decide():
        if journal_decision and "decide" in journal: return journal["decide"]
        d = stub_model("T-77")
        if journal_decision: journal["decide"] = d
        return d
    first = decide()                         # the process acts on this, then dies before finishing
    second = decide()                        # the restarted process needs the decision again
    return first, second

for flag in (False, True):
    f, s = run_to_crash_and_resume(flag)
    print(f"decision journaled={flag!s:5}  before crash: {f['action']:13} after restart: {s['action']:13} same: {f == s}")

decision journaled=False  before crash: refund        after restart: store_credit  same: False
decision journaled=True   before crash: refund        after restart: refund        same: True

Without a journaled decision, the process acted on “refund” before the crash and, after the restart, asked again and received “store_credit”. With the decision recorded as a step result, the restart reads it back. The practical rule is that every non-deterministic input to the workflow (a model output, a clock reading, a random value, an external lookup used to branch) is recorded when first obtained and reused on replay. The planning state from the previous article belongs in the journal for the same reason, along with the counters behind its budgets: a budget held in memory resets at every restart and stops bounding anything.

Durable execution engines

Writing a journal and a replay loop by hand is instructive and rarely what a production system should do. Workflow engines package the same ideas. Temporal’s documentation describes workflow code that is replayed from its event history after a failure and must make the same decisions each time, with API calls, database queries, and LLM invocations placed in activities whose recorded result is reused during replay. Microsoft’s documentation for Durable Functions and the Durable Task SDKs states the same constraint, that orchestrator code must be deterministic because it replays, and lists what to avoid there: direct network calls, wall-clock reads, random values, and blocking sleeps, which the engine offers durable replacements for. The glossary entry on durable execution summarizes the model.

What the engines provide is the journal, the replay, durable timers for long waits, and retry policies. What they do not provide is safety for the effects themselves. Temporal’s documentation says activities may be retried and may therefore execute more than once, and recommends that they be idempotent. A worker that dies after an activity’s effect but before the engine records its completion is the third crash point of the table above, and the engine resolves it by running the activity again. Making that rerun harmless depends on keys, lookups, and idempotent operations, which the following sections develop.

One more case needs no engine. When the effect is a row in the same database as the journal, the effect and the journal record can commit in one transaction, and the crash window disappears. The transactional outbox applies the same idea to messages. Most agent tools reach a different system, so the window remains the usual case.

Duplicate effects

Run 2 above survived because the key was stable. This section measures how much that matters, first with a probability model and then by breaking the executor at every point.

How often blind retries duplicate

Suppose each attempt on a step ends as success with probability \(s\), unavailable with probability \(f\), timeout with the effect applied with probability \(u\), or timeout with the effect lost with probability \(v\), where \(s + f + u + v = 1\). A retry loop repeats after every non-success up to \(N\) attempts, and each retry carries a fresh key, so the service sees a new operation each time.

An effect is duplicated when at least two attempts were applied. Applied attempts are successes and applied timeouts, but a success ends the loop, so the only applied outcome that lets the loop continue is \(u\). A duplicate therefore needs one applied timeout followed by at least one more applied attempt. Write \(q = f + v\) for the non-applied failures that also continue the loop, and let \(A_m\) be the probability that at least one of the next \(m\) attempts is applied, so \(A_m = (s + u) + q\,A_{m-1}\) with \(A_1 = s + u\). Then the duplicate probability \(P_N\) satisfies

\[ P_N = u\,A_{N-1} + q\,P_{N-1}, \qquad P_1 = 0 . \]

In words: the first attempt is an applied timeout with probability \(u\) and then any later applied attempt completes the duplicate, or it is a non-applied failure with probability \(q\) and the question restarts with one fewer attempt. For two attempts, \(P_2 = u(s+u)\). For three, \(P_3 = u(s+u)(1+2q)\).

The following program evaluates the recursion and checks it against a simulation. The rates are illustrative values chosen for the example, not measurements: 97% success and 1% for each of the other three outcomes.

"""Duplicate-effect probability for blind retries. Python 3.12, standard library only."""
import random

# One attempt ends in one of four ways. 'applied' says whether the effect happened at the service.
OUTCOMES = {"success": (0.970, True), "unavailable": (0.010, False),
            "timeout_applied": (0.010, True), "timeout_lost": (0.010, False)}

def exact(attempts_left, effects=0):
    """Blind retry of every non-success, with a fresh key each time: P(two or more effects)."""
    dup = 0.0
    for outcome, (p, applied) in OUTCOMES.items():
        e = effects + applied
        if outcome == "success" or attempts_left == 1:
            dup += p * (e >= 2)                          # the loop stops here
        else:
            dup += p * exact(attempts_left - 1, e)       # retry after a failure or timeout
    return dup

def simulate(max_attempts, trials, rng):
    names, probs = list(OUTCOMES), [OUTCOMES[o][0] for o in OUTCOMES]
    dups = 0
    for _ in range(trials):
        effects = 0
        for _ in range(max_attempts):
            o = rng.choices(names, probs)[0]
            effects += OUTCOMES[o][1]
            if o == "success": break
        dups += effects >= 2
    return dups / trials

s, u = OUTCOMES["success"][0], OUTCOMES["timeout_applied"][0]
print(f"closed form for 2 attempts: u*(s+u) = {u * (s + u):.5f}")
for n in (2, 3):
    print(f"{n} attempts: exact {exact(n):.5f}   simulated {simulate(n, 400_000, random.Random(7)):.5f}")
print(f"per 100,000 refunds with 3 attempts: about {exact(3) * 100_000:.0f} double refunds without a stable key, 0 with one")

closed form for 2 attempts: u*(s+u) = 0.00980
2 attempts: exact 0.00980   simulated 0.00993
3 attempts: exact 0.01019   simulated 0.01029
per 100,000 refunds with 3 attempts: about 1019 double refunds without a stable key, 0 with one

The recursion and the simulation agree: about 1.0% of steps end with two effects when the retry reuses nothing. The count is driven by \(u\), the rate of applied timeouts, and barely moves with \(N\): going from two attempts to three raises the figure from 0.98% to 1.02%, because a third attempt only matters after two failures in a row. Lowering the attempt limit does not fix the problem, and setting it to one gives up the benefit of retrying transient failures. With a stable key, a repeated attempt returns the first result, so the number of effects per step is at most one regardless of \(u\) and \(N\). At 1% applied timeouts, 100,000 refunds would contain about a thousand double refunds without a key, which is why a key on a money-moving step is a requirement and not an optimization.

Which mechanism removes which duplicate

The formula covers retries. Crashes add the windows from the earlier table. The experiment below runs the executor with a crash at every tick in turn, lets a new process resume, and counts how many of those crash points end with two or more refunds, labels, or emails. It repeats the sweep for four configurations that add one mechanism at a time:

• A: no journal and fresh keys. A restart begins again from the first step.
• B: a journal but fresh keys. The restart knows which steps finished but generates a new key for the step that was in progress.
• C: a journal and stable keys.
• D: as C, and an unknown outcome on the email step is resolved by looking for the email.

A second sweep adds scripted faults: the refund response and the first email response are lost after the effect was applied.

from engine import *
INP = {"order": "ord_1002", "cents": 4200, "ticket": "T-77", "customer": "cu_ana"}
CONFIGS = {
    "A  no journal, fresh keys":           dict(journal=False, stable_keys=False, lookup=False),
    "B  journal, fresh keys":              dict(journal=True,  stable_keys=False, lookup=False),
    "C  journal, stable keys":             dict(journal=True,  stable_keys=True,  lookup=False),
    "D  journal, stable keys, lookup":     dict(journal=True,  stable_keys=True,  lookup=True),
}

def attempt(flags, faults, crash_at):
    db, w = new_journal(), World(faults); start(db, "wf1", INP)
    ex = Executor(db, w, "wf1", crash_at=crash_at, **flags)
    try:
        ex.run(INP)
    except Crash:
        ex = Executor(db, w, "wf1", **flags); ex.run(INP)      # a new process; journal=False loses its memory
    return db, w, ex

def sweep(faults):
    print(f"{'configuration':34}{'crash points':>13}{'2+ refunds':>12}{'2+ labels':>11}{'2+ emails':>11}{'not completed':>15}")
    for name, flags in CONFIGS.items():
        _, _, probe = attempt(flags, faults, None)                # a run with no crash, to count the tick points
        n = probe.ticks
        bad = {"refunds": 0, "labels": 0, "emails": 0, "open": 0}
        for k in range(1, n + 1):
            db, w, ex = attempt(flags, faults, k)
            bad["refunds"] += len(w.refunds) > 1
            bad["labels"] += len(w.labels) > 1
            bad["emails"] += len(w.emails) > 1
            bad["open"] += ex.status() != "completed"
        print(f"{name:34}{n:>13}{bad['refunds']:>12}{bad['labels']:>11}{bad['emails']:>11}{bad['open']:>15}")

print("crash at every tick, no service faults")
sweep({})
print("\ncrash at every tick, plus a lost refund response and a lost email response")
sweep({("issue_refund", 1): "timeout_after", ("send_email", 1): "timeout_after"})

crash at every tick, no service faults
configuration                      crash points  2+ refunds  2+ labels  2+ emails  not completed
A  no journal, fresh keys                    20          10         18          6              0
B  journal, fresh keys                       20           1          1          1              0
C  journal, stable keys                      20           0          0          1              0
D  journal, stable keys, lookup              20           0          0          0              0

crash at every tick, plus a lost refund response and a lost email response
configuration                      crash points  2+ refunds  2+ labels  2+ emails  not completed
A  no journal, fresh keys                    22          22         20         22              0
B  journal, fresh keys                       22          22          1         22              0
C  journal, stable keys                      22           0          0         22              0
D  journal, stable keys, lookup              21           0          0          0              0

In A, a restart repeats everything: 18 of 20 crash points leave two labels, and any crash after the refund call leaves two refunds, 10 of 20. In B, the journal fixes most of it, since finished steps are skipped, but the crash window after a call still duplicates, once per effect step, because the resumed attempt sends a fresh key for an operation that already happened. In C, stable keys remove those duplicates for the refund and the label. The email still duplicates at one crash point, because the email service accepts no key and a repeated send is a new send. In D, looking for the email before sending removes the last duplicate.

The second table adds lost responses, and it separates the two kinds of failure. The lost-response faults produce duplicates at every crash point in A and B, since the retry after a timeout uses a fresh key whatever the crash does. C removes the refund and label duplicates and cannot touch the emails, which duplicate in all 22 runs. D removes them by looking for the email after the timeout. D also finishes in 21 ticks instead of 22, because the email step needs one attempt fewer. No configuration ends with an unfinished workflow in this experiment, so the differences are all duplicates.

The sweep covers one toy workflow, a single crash per run, and two scripted faults. It shows that each mechanism fixes a specific class of duplicates; it does not show that D is safe in general.

What makes a key work

A key only helps if the receiving system honors it, and the conditions are properties of that system.

• The effect and the key are recorded together. The AWS Builders’ Library says that recording the client request identifier and performing the mutation should be one atomic operation, so that a crash between them cannot leave an applied effect with no record. In the fake, _keyed does this by construction.
• The same key with different parameters is rejected. Both the AWS article and Stripe’s documentation describe comparing the new request’s parameters with the original and failing on a mismatch. This is what makes it safe to derive a key from the step identity: if the model changes the refund amount, the step is a different logical operation and needs a new key, for example by including a revision number in the step name.
• The key outlives the retry horizon. Stripe’s documentation says keys can be removed once they are at least 24 hours old, after which a reused key creates a new request. A workflow that may keep retrying a step for days needs either a shorter retry window or a different guard, such as a lookup.
• Error handling is part of the contract. Stripe saves the status and body of the first request for a key whether it succeeded or failed, including 500 errors, and returns the same result on a repeat; it does not save a result when validation failed or a concurrent request held the key. So retrying a 500 with the same key replays the 500 on that provider. Whether a repeat after an error re-runs the operation, replays the error, or reports “in progress” differs by provider and has to be read from its documentation. The idempotency key entry lays out the states a server’s record can be in.

For systems that offer no key, the options in order of preference are a naturally idempotent operation (voiding a label twice is voiding once), a lookup that finds the earlier effect before repeating (the email case), a deliberate decision to accept a rare duplicate where the cost is small and the effect is visible, and, if none of those apply, not repeating automatically at all and treating the unknown state as a case for a person. A lookup followed by a send is a check-then-act sequence, and two workers can both pass the check; the next subsection prevents that.

Two workers on one workflow

A durable workflow needs exactly one worker at a time, but workers stall, and a stalled worker can look dead. The common arrangement is a lease: a worker claims the workflow for a time-to-live, and another worker may take it after the lease expires. A paused worker that wakes up after losing its lease must not keep writing. Each claim therefore increments a generation number, and every journal write checks that its generation is still current. This is the fencing idea, and in the executor above it is the claim method together with _fence.

The next program stalls worker A right after the refund call, lets B try too early, lets B take over after the lease expires, and then has A attempt a delayed journal write.

from engine import *
INP = {"order": "ord_1002", "cents": 4200, "ticket": "T-77", "customer": "cu_ana"}
now = [0.0]
db, w = new_journal(), World(); start(db, "wf1", INP)

a = Executor(db, w, "wf1", clock=lambda: now[0], crash_at=11)   # worker A stalls just after the refund call
a.claim(ttl=30)
try: a.run(INP)
except Crash: print("A stalls after calling the refund service (generation", a.gen, ")")

b = Executor(db, w, "wf1", clock=lambda: now[0])
now[0] = 5.0
try: b.claim(ttl=30)
except Busy: print("B tries at t=5: lease still held -> Busy")
now[0] = 45.0
b.claim(ttl=30); print("B claims at t=45: generation", b.gen)
print("B runs the workflow:", b.run(INP))

try: a.put("issue_refund", "done", 1, {"refund": 1})            # A wakes up and tries its delayed journal write
except Fenced as e: print("A's delayed write is rejected:", e)
print("refunds:", len(w.refunds), "emails:", len(w.emails), "status:", b.status())

A stalls after calling the refund service (generation 1 )
B tries at t=5: lease still held -> Busy
B claims at t=45: generation 2
B runs the workflow: completed
A's delayed write is rejected: generation 1 is stale
refunds: 1 emails: 1 status: completed

B is refused while A’s lease holds, claims at the later time with a new generation, and completes the workflow with one refund and one email. A’s delayed write is rejected because its generation is stale. Two defenses did two jobs: the stable key stopped B’s repeat of the refund from creating a second one, and the fence stopped A from corrupting the journal afterward. A lease alone cannot guarantee that A has stopped acting; it can only bound what A may record, which is why the key still has to hold.

Uncertain outcomes

An intent without a result is unknown, and “unknown” is neither failure nor success. It must not be reported as either.

A step therefore has more than two terminal states. In the journal here they are done, failed (the service refused it, or every attempt was a clean failure), and unknown (an attempt timed out and nothing resolved it). The distinction reaches the model: a tool result of “failed” invites the agent to try again, possibly with a new key, which is how one unknown outcome becomes a duplicate. The authorization article makes the same point from the dispatcher’s side: an unknown result should be reported to the model as unknown, with an instruction that the runtime will reconcile it and that no new attempt should be started.

Resolving an unknown step has four options, ordered by how little they assume.

1. Repeat with the same key. If the service honors keys, this is both a retry and a reconciliation. The repeat returns the first result if the effect happened and performs it if it did not. This is what happened to the refund in run 2.
2. Look for the effect. If the service has a lookup by a reference you control, search first. This is the email path: the unknown send is resolved by find_email, and the send is repeated only if nothing is found.
3. Wait and reconcile. Some effects appear later: a queued job, an asynchronous payment. A reconciliation job revisits unknown steps on a schedule and asks the service for the current state of the operation.
4. Escalate. When none of the above is available, the workflow stops with the step marked unknown and hands the case to a person with the journal as evidence.

Two runs of the executor show the extremes. Run 5 loses the response to the first email send after it was applied. Run 7 makes label creation time out on every attempt.

scenario("5. email applied but the response is lost", {("send_email", 1): "timeout_after"})
scenario("7. label creation times out on every attempt", {("create_label", n): "lost" for n in (1, 2, 3)})

5. email applied but the response is lost
  journal : create_label=done(1), schedule_pickup=done(1), issue_refund=done(1), send_email=done(1), close_ticket=done(1)
  status  : completed
  world   : refunds=1 emails=1 labels=['active'] pickups=['scheduled'] tickets={'T-77': 'closed'}
7. label creation times out on every attempt
  journal : create_label=unknown(3)
  status  : escalated (create_label: outcome unknown)
  world   : refunds=0 emails=0 labels=[] pickups=[] tickets={}

In run 5 the journal shows send_email=done(1) and the world holds one email. The first send applied and timed out; the lookup found it, so the step finished without a second send and without a second attempt. In run 7, three timeouts exhaust the attempt limit with no way to tell whether a label exists. The step is unknown, the workflow is escalated, and nothing was compensated, deliberately: the label may exist, and compensating a step whose result (the label ID) was never received is not possible. In this world nothing was applied, but the executor cannot know that.

Escalation does not have to be final. A reconciliation job can later repeat the unknown step with its original key. If the service answers, the step becomes done with its result, and the workflow can continue or compensate with full information. The executor above does not implement that job; it stops at the escalation.

Retry conditions

The previous sections used retries without saying when to use them. A durable workflow has an unusual retry problem: steps can be repeated over minutes or days, repeats cross process restarts, and an automatic repeat after an unknown outcome is exactly the dangerous one. The article on model API integration covers backoff, jitter, deadlines, and retry amplification for a call to a model. The question here is narrower: given a failure of a side-effecting step, repeat or not?

Three facts decide it: the error class, how the step can safely be repeated, and the remaining budget. The decision fits a table, and writing it as data keeps the policy reviewable and testable.

"""Retry decision as data: (error class, how the step can be repeated) -> action. Standard library only."""
# How a step can be repeated:  keyed   = the service dedupes on a key we control
#                              lookup  = no key, but we can search for the effect before repeating
#                              natural = repeating has the same effect by construction (void, close)
#                              none    = repeating may duplicate and the effect cannot be found
def decide(error, repeat, attempts, max_attempts=3):
    if error in ("rejected", "invalid", "unauthorized"):
        return "fail_step"                                # retrying cannot change a permanent answer
    if attempts >= max_attempts:
        return "escalate" if error == "timeout" else "fail_step"
    if error in ("unavailable", "rate_limited"):
        return "retry"                                    # the request was not applied
    if error == "timeout":                                # applied or not: unknown
        return {"keyed": "retry_same_key", "natural": "retry", "lookup": "lookup_then_retry", "none": "escalate"}[repeat]
    raise ValueError(error)

errors = ["unavailable", "rate_limited", "timeout", "rejected"]
print(f"{'error':14}" + "".join(f"{r:>20}" for r in ("keyed", "lookup", "natural", "none")))
for e in errors:
    print(f"{e:14}" + "".join(f"{decide(e, r, 1):>20}" for r in ("keyed", "lookup", "natural", "none")))
print("third attempt, timeout, keyed:", decide("timeout", "keyed", 3), "| third attempt, unavailable:", decide("unavailable", "keyed", 3))

error                        keyed              lookup             natural                none
unavailable                  retry               retry               retry               retry
rate_limited                 retry               retry               retry               retry
timeout             retry_same_key   lookup_then_retry               retry            escalate
rejected                 fail_step           fail_step           fail_step           fail_step
third attempt, timeout, keyed: escalate | third attempt, unavailable: fail_step

The rows read as follows. An error that says the request was not applied (unavailable, rate_limited) is safe to repeat for any step. A permanent answer (rejected, invalid, unauthorized) is never repeated, since a different answer will not come from the same request. The timeout row is where the step’s repeatability decides: repeat with the same key if the service honors one, look first if there is a lookup, repeat plainly only if the operation is naturally idempotent, and escalate if repeating could duplicate and nothing can detect it. The last line shows the budget: when attempts are exhausted, a timeout escalates, because it is unknown, while a clean failure fails the step.

Several details turn this table into something that holds up across restarts.

Count attempts in the journal, before the call. The executor writes the attempt number before it calls. If a step crashes the worker every time it runs, a counter held in memory restarts at zero on each restart and the workflow loops forever; a counter written before the call reaches the limit across restarts and the workflow stops. This is the budget idea from the previous article applied to retries.

Retry in one layer. An engine’s activity retry policy, the HTTP client’s retries, and the model SDK’s retries multiply: three layers of three attempts make nine calls per step, and the duplicate probability above applies to every one of them. Choose the layer that sees the step’s identity and key, which is the durable layer, and turn the others off for side-effecting calls.

Match the retry window to the guard. If the step is protected by a key that expires after a day, the workflow’s retry horizon for that step must be shorter than a day, or the policy must switch to a lookup before the key expires.

Separate retrying from re-planning. A refund the service rejected because it exceeds the refundable amount is not retried. The runtime can report the rejection to the model, as in the authorization article, and the model may propose a different action, such as a smaller refund. That is a new decision and a new step with a new identity, subject to the planning limits and budgets of the previous article. It is not a retry of the old step.

Wait durably. A backoff of an hour is not a sleep in a worker. Engines offer durable timers for this; in a hand-built system the workflow stores a retry_at time and a poller picks it up. The executor here omits waiting to keep the experiments instant.

Compensation and recovery

Retries repeat work. Compensation undoes it. When a step fails permanently after earlier steps committed, there is no transaction to roll back: each earlier step committed in its own system. The workflow has to perform new business operations that reverse the effects. The pattern is the saga, introduced by Garcia-Molina and Salem in 1987 for long-lived transactions, and the glossary entries on saga and compensating transaction give the background.

Classify each step

Microsoft’s description of the saga pattern sorts steps into three kinds, and applying it to the workflow shows where the pivot sits:

Step	Kind	If it fails	If a later step fails
create a return label	compensable	repeat with the same key	void the label
schedule a carrier pickup	compensable	repeat with the same key	cancel the pickup
refund 42.00 USD (pivot)	the commitment	compensate earlier steps	no compensation; move forward
email the customer	retryable, after the pivot	repeat, then escalate	not undone
close the ticket	retryable, after the pivot	repeat, then escalate	not undone

The pivot is where the workflow commits to completion. Before it, the effects are cheap to reverse, so a permanent failure sends the workflow backward. After it, the refund has left the system, so a failure sends the workflow forward: the remaining steps are retried until they succeed or a person takes over. The Azure Architecture Center’s pattern on compensating transactions advises designing the workflow so that irreversible steps occur only after all critical validations succeed, which is why the refund comes after the label and the pickup are secured and not before.

What a compensation is

A compensation is a new forward operation that reverses the business effect, and it is not a restore of an earlier state. Voiding a label leaves a voided label in the label system, which is a correct and auditable record; deleting the label row would erase evidence and break anything that already referenced it. A pickup is canceled, not removed. Microsoft’s pattern page adds that compensation need not run in strict reverse order and need not return the system to what it was before the operation, since other work may have happened in between.

Four requirements follow from the fact that compensations run when something has already gone wrong.

• A compensation needs the same machinery as a forward step. It must be journaled, keyed, retried, and resumable. In the executor, the undo steps are executed by the same execute method, under journal names such as undo:create_label, so a crash during compensation resumes the compensation and does not start over.
• A compensation must be safe to repeat. The fake void_label is naturally idempotent: voiding an already voided label changes nothing, and voiding a label that does not exist is a no-op.
• A compensation can fail. The design has to say what happens next.
• A compensation needs inputs from the step it reverses. The undo of create_label needs the label ID, which is in the journal only if the step finished. That is why the executor compensates only steps in the done state and escalates unknown ones.

Two runs show the compensation path. In run 3, the refund is rejected as permanently impossible. In run 4, the refund is rejected and the carrier’s pickup cannot be canceled.

scenario("3. refund rejected: compensate the label and the pickup", {("issue_refund", 1): "reject"})
scenario("4. refund rejected and the pickup cannot be cancelled", {("issue_refund", 1): "reject", ("cancel_pickup", 1): "reject"})

3. refund rejected: compensate the label and the pickup
  journal : create_label=done(1), schedule_pickup=done(1), issue_refund=failed(1), undo:schedule_pickup=done(1), undo:create_label=done(1)
  status  : compensated (issue_refund failed)
  world   : refunds=0 emails=0 labels=['void'] pickups=['cancelled'] tickets={}
4. refund rejected and the pickup cannot be cancelled
  journal : create_label=done(1), schedule_pickup=done(1), issue_refund=failed(1), undo:schedule_pickup=failed(1), undo:create_label=done(1)
  status  : escalated (issue_refund failed; could not run cancel_pickup)
  world   : refunds=0 emails=0 labels=['void'] pickups=['scheduled'] tickets={}

Run 3 shows the happy compensation. The journal ends with undo:schedule_pickup=done(1), undo:create_label=done(1), in reverse order, the label is void, the pickup is cancelled, and the status is compensated. The refund never happened, so the world has no refund and no email. Run 4 shows the failing compensation. The pickup cancellation was rejected, so the workflow still voided the label, which was independent, and ended escalated with the note naming the cancellation that could not run. The world is now inconsistent in a specific, recorded way: a pickup remains scheduled for a voided label. The system cannot repair that automatically, and the right behavior is to preserve the evidence and hand it to someone who can call the carrier.

After the pivot

Run 6 shows the forward direction. The refund succeeded, then the email service was unavailable for every attempt.

scenario("6. email service down for every attempt", {("send_email", n): "unavailable" for n in (1, 2, 3)})

6. email service down for every attempt
  journal : create_label=done(1), schedule_pickup=done(1), issue_refund=done(1), send_email=failed(3)
  status  : escalated (send_email failed after the pivot)
  world   : refunds=1 emails=0 labels=['active'] pickups=['scheduled'] tickets={}

The refund stands, the journal records send_email=failed(3), and the workflow is escalated with the reason. There is nothing to compensate: voiding the label and pickup would leave a refunded customer without instructions for returning the item. The correct recovery is forward, which means retrying the email later, perhaps through a different channel, and closing the ticket. A person or a scheduled job picks it up with the journal showing exactly which steps are done.

What compensation cannot undo

Compensation reverses effects in systems the workflow controls. It does not erase what someone observed or what depends on the step. A confirmation email that was sent stays sent; the best a compensation can do is send a correction, which is another effect with its own risks. A customer who saw “your pickup is scheduled” saw it. And because each step commits on its own, other workflows can observe intermediate states: a second claim on the same order, started while the first is between its label and its refund, sees a label that may be voided in a moment. The saga pattern gives no isolation across steps, and the Azure documentation lists countermeasures such as semantic locks, re-reading values before acting, and ordering steps so that the visible state changes late. In this workflow, the order-state check that the tool layer already performs before a refund serves as the re-read, and a per-order lock on the claim could serve as the semantic lock.

Testing recovery by crashing everywhere

The sweep earlier is also the test method for the recovery logic. Enumerate every point where the process can die, kill it there, resume, and assert invariants about the world: at most one refund per order, at most one label, every workflow ends in completed, compensated, or escalated, and never in running with nobody holding it. A hand-written test for “the process dies after the refund” covers one point of twenty and tends to be the point the author already thought about. The sweep also finds the points they did not: the configuration D row of the first table has no duplicates, and the same sweep showed that C still duplicates one email, which a single hand-picked crash point is unlikely to have found.

The invariants can be extended with the faults of the second table, and the sweep can be repeated with each fault on each call. The evaluation of whole runs, including adversarial and failure cases, is the subject of Evaluating Agent Actions and Workflow Outcomes.

Handing a stuck workflow to a person

Every escalated status above has the same shape: a workflow stopped in a defined state, with a reason, a journal, and a world that may need manual repair. What a person needs in order to act is the list of completed steps with their results, the step that failed or is unknown with its attempts, the compensations that ran and the ones that did not, and the effects that cannot be undone. The journal holds all of that. How the approval or escalation request is presented, who may decide, what happens if the world changes while a person is deciding, and how long a decision may wait are the subject of Human Approval and Escalation in AI Workflows.

Where this design stops

A durable workflow guarantees that the process remembers what it did and does not repeat effects it can detect. It does not guarantee that what it did was right. A journaled decision to refund the wrong order is executed exactly once, durably, and compensation will not question it; the checks of the authorization article and the evaluation of agent actions are what stand between a bad decision and a durable one.

Exactly-once is not a property of any one component here. What the design gives is at-least-once attempts combined with idempotent or detectable effects, which yields one effect per logical step as long as the key’s scope, retention, and atomicity assumptions hold. A service with no key, no lookup, and no natural idempotency forces a choice between a possible duplicate and a possible omission, and no executor removes that choice.

Long-lived workflows outlive code. Microsoft’s documentation warns that changes to orchestrator code can break the replay of workflows already running and recommends versioning strategies such as side-by-side deployment. A hand-built executor that replays by step name has the same exposure when step names, their order, or their result shapes change. The journal also stores customer data and model outputs, so its retention and access controls are part of the privacy design.

Finally, the sketch here omits what a production system adds: durable timers, a reconciliation job for unknown steps, parallel steps, sub-workflows, cancellation, and observability that ties each journal row to a trace. Those are reasons to adopt an engine and keep the lessons above: keep decisions and results in a journal, derive keys from step identity, treat an intent without a result as unknown, decide retries from error class and repeatability, and design the reversal of each step before the step ships.

References

• Stripe, “Idempotent requests” (Stripe API reference): how keys are stored, what is saved for failed requests, key retention, and parameter comparison.
• Amazon Builders’ Library, “Making retries safe with idempotent APIs”: client request tokens and recording the token atomically with the mutation.
• Temporal documentation, “Activities”: at-least-once execution of activities and the recommendation that they be idempotent.
• Temporal documentation, “Workflows”: deterministic workflow code, event history replay, and why side effects belong in activities.
• Microsoft Learn, “Durable orchestrator code constraints” (Durable Functions and Durable Task SDKs): determinism, replay, durable timers, and versioning of running orchestrations.
• Microsoft Azure Architecture Center, “Saga design pattern”: compensable, pivot, and retryable transactions; choreography and orchestration; data anomalies and countermeasures.
• Microsoft Azure Architecture Center, “Compensating Transaction pattern”: compensation as application-specific reversal, idempotent and resumable compensation, and points of no return.