Agent Runtime Isolation and Execution Policies
Consider a hypothetical platform that hosts customer-support agents for many companies, called tenants. An agent reads a ticket, looks up the order, and issues a refund through a payments API. In one run the payments call succeeds but the response is lost. The runtime retries, which refunds the customer twice. In another run the text of a ticket, written by an outsider, tells the agent to look up an order number that belongs to a different tenant. The agent holds a platform-wide service key, so the lookup works. Neither failure involves a bad model answer in the usual sense. The first is a retry that repeats a side effect, and the second is a credential and a rule that did not distinguish between tenants. Both were the runtime’s job to prevent.
An agent is a loop in which a model proposes an action, a program executes it, and the result is fed back for the next proposal. The AI Agent glossary entry describes the autonomy boundary this implies: the available tools, the permitted resources, the approvals required for consequential actions, and the limits on time, cost, and repeated attempts. On a shared platform, the component that enforces that boundary is the agent runtime. This article explains what the runtime must enforce and why each control sits where it does, and then implements the core of it, a policy engine, approvals, idempotent execution, resumable run state, short-lived credentials, and a tamper-evident action log, in a small program that prints what it does. By the end you should be able to choose an isolation tier for a class of agent work, explain delegation versus service authority for tool credentials, write execution rules that hold regardless of what the model says, bind an approval to one exact action, define what counts as “the same action” for retries, and describe what a platform must record about every action.
The runtime, services, and tokens in the code are written for this article in Python’s standard library (Python 3.12, macOS) and run in one process. They illustrate control flow under stated assumptions. The token format is a toy signed token, the signing key is a placeholder, and no real service or model is called. Product and standard behavior is described from public documentation listed at the end.
What this article builds on
Prompt and Model Promotion with Evaluation Gates decides whether a new prompt or model may start serving. A gate cannot make an agent safe to run, since evaluation sets are finite and a harmful action happens at runtime, so the controls here operate while the agent is live. Operating Coding Agents with Restricted Permissions covers a single developer’s coding agent: operating-system sandboxes, argument-level tool rules, and approvals for a shell and a checkout. That article’s results carry over and are summarized where they are used. This article takes the other situation, a platform that runs many tenants’ agents as a service, where the new problems are separating tenants from each other, giving each action an identity, and surviving retries and restarts.
What the runtime enforces
The core rule of an agent runtime is that the model proposes and the runtime disposes. The model’s output is a request to take an action, expressed as text. Anything that can influence the model, such as a document it reads, a tool result, or a ticket written by a customer, can influence that request. Prompt Injection explains why content must not grant itself authority: the order service must still enforce access restrictions even if the model repeats a sentence that tells it to ignore them. The same reasoning applies to every control below. A rule written in the prompt shapes what the model tries to do. A rule enforced by the runtime decides what happens.
OWASP’s 2025 entry on excessive agency names three root causes of damaging actions by an LLM-based application: excessive functionality (tools the task does not need), excessive permissions (tools or identities that can do more than the task needs), and excessive autonomy (high-impact actions without a human check). Its mitigations include minimizing the extensions an agent can call and their functionality, avoiding open-ended tools such as a general shell where a narrow function would do, executing in the context of the specific user with minimal privileges, requiring human approval for high-impact actions, and complete mediation, which means enforcing authorization in the downstream system and not relying on the model to decide whether an action is allowed. The Model Context Protocol (MCP) specification reaches compatible conclusions for tools: it says there SHOULD always be a human in the loop with the ability to deny tool invocations, and that clients must treat tool annotations, the descriptive metadata a server attaches to a tool, as untrusted unless they come from a trusted server.
The runtime is the place where those principles become code. A tool call passes through six steps, each of which is a section below.
model proposes: tool + arguments
-> normalize arguments (so equivalent requests look identical)
-> policy decision (ALLOW / REQUIRE_APPROVAL / DENY, with a rule id)
-> approval check, if required (bound to this exact call, single use)
-> mint a short-lived credential (this tenant, this tool, this audience)
-> execute in an isolated environment, with an idempotency key
-> record state and an audit entryIsolation for a multi-tenant runtime
Two things run in a runtime, and they need different isolation. The agent loop is platform code: it calls the model and applies policy. The tool executions are where untrusted behavior shows up, because a tool may run code the model wrote, process files the user uploaded, or call out over the network with arguments the model chose. Isolation primarily protects tenants from each other and the platform from the tools.
The coding-agent article draws the distinction between a permission gate, which reasons about what the agent asked for, and an isolation boundary, which restricts what a running process can reach whatever it asked for. In a platform, the gate is the policy engine in this article, and the boundary is the execution environment. A gate has to understand the request. A boundary only knows resources, so it enforces coarse facts: which files are writable, which hosts are reachable, how much memory and time the process may use. The hard guarantees belong in the boundary.
The choice of boundary is a choice of how much of the host’s kernel is shared with the tool, and the options are well documented.
| Tier | What isolates | What is shared | Typical use |
|---|---|---|---|
| Process with limits | Operating-system user, resource limits, file permissions, a scrubbed environment | The whole host kernel | Calling a narrow, allowlisted API where no model-written code runs |
| Container | Namespaces, control groups, optional syscall filters | The host kernel | Trusted code with a per-run filesystem and network policy |
| User-space kernel (gVisor) | A separate application kernel per sandbox that implements the Linux system-call interface in a memory-safe language, so containers interact with it instead of the host kernel | Only the limited set of host interfaces the application kernel itself uses | Running untrusted or model-written code at lower cost than a virtual machine |
| MicroVM (Firecracker) | Hardware virtualization with a minimal device model; the project’s stated mission is secure, multi-tenant, minimal-overhead execution of container and function workloads | Only the hypervisor and hardware | Code from mutually untrusting tenants where a kernel exploit must not cross tenants |
On Kubernetes, the RuntimeClass resource (stable since v1.20) lets different pods use different container runtime configurations, so a platform can place model-written code in a stronger tier and ordinary tool calls in a cheaper one on the same cluster. Its documentation describes the motivation as balancing performance and security, and notes that stronger runtimes cost extra overhead.
Choosing the tier follows the consequence of an escape, not the likelihood. If a tool only calls an allowlisted API with arguments validated by the runtime, a process boundary with a scrubbed environment may suffice. If a tool runs code the model wrote, or parses untrusted file formats, a kernel exploit becomes a realistic route between tenants and a user-space kernel or microVM is warranted. Several rules hold across tiers:
- Give each run an ephemeral workspace that is created for it and deleted afterward, and never a writable volume shared between tenants.
- Deny network egress by default and allow specific destinations. Egress is how data leaves.
- Keep credentials out of the environment. A sandbox restricts system calls, and the coding-agent article’s experiment showed that an environment variable remains visible inside a sandbox that otherwise blocks file reads and network connections.
- Set limits on wall-clock time, CPU, memory, and process count, and kill what exceeds them.
- Remember what isolation does not provide. Sandbox makes the point that a lookup tool inside a sandbox can still disclose records if its credential permits reading every tenant. Isolation limits reach, and authorization still has to say what each caller may request. Workload Isolation separates the goals that isolation can serve: contention, failure propagation, access, and cost, and a boundary built for one rarely delivers the others.
Tool credentials: whose authority does a call carry?
Every call from an agent to a downstream service carries some authority, and the platform decides whose. Three designs exist.
- A shared service credential. The runtime holds one key per downstream service and uses it for every tenant and user. It is simple, and it makes the runtime a confused deputy, a privileged intermediary that can be induced to use its power on behalf of someone who does not have it. In the opening story, a platform-wide key made the cross-tenant lookup succeed. The only thing separating tenants is the runtime’s own checks.
- A per-tenant service credential. Each tenant gets its own credential for each service. It stops one tenant from reaching another through the downstream service, but within a tenant every user’s agent has the same power.
- Delegated, short-lived credentials. The runtime obtains a token for each tool call that identifies the user on whose behalf the agent acts, the agent acting, the tenant, the intended service (audience), and the scopes, and that expires in seconds or minutes. The downstream service validates all of it.
The third is the one that matches least privilege, and it has a standard vocabulary. The OAuth 2.0 Token Exchange specification (RFC 8693) describes a service that calls a backend on behalf of a user, and distinguishes delegation, in which the caller uses its own credentials along with an annotation that it is acting on behalf of the user, from impersonation, in which the caller gets a limited credential that continues to identify the user. It defines claims for expressing the acting party. The demo token follows the delegation idea: sub is the user, act is the agent, and the token also carries the tenant, an audience, scopes, and an expiry.
MCP’s authorization specification gives two further rules that apply to any runtime that sits between a client and a downstream API. A protected server is an OAuth resource server, and it must validate that a token was issued for it, for example through the audience claim. And “token passthrough”, accepting a token from a client and forwarding it unchanged to a downstream API, is explicitly forbidden, because it breaks the audience boundary and lets the downstream service believe the token was validated by the runtime. The credential the runtime presents downstream should be one it obtained for that downstream service.
The model never sees any of this. The runtime’s credential broker mints the token at dispatch time, uses it for one call, and discards it. Tool results returned to the model should be checked so that they do not contain credentials. The following module of this course covers redaction in logs and traces.
Tools and execution policies
A tool registry describes each tool with facts the runtime relies on and the model does not control. The demo registry records, for each tool, the kind of effect it has (read, write, or send), the scope it requires, how repeated calls are recognized as the same action, and whether the downstream service can deduplicate. The model supplies only the arguments. Everything else is declared by the platform.
The policy engine takes the run (tenant, user, granted tools, delegated scopes, budget), the tool, and the normalized arguments, and returns one of three decisions with the identifier of the rule that produced it.
- Default deny. A call is allowed only after every check passes. A tool that was not granted to this agent is refused before anything else is examined.
- Authorization comes from the run, not from the model’s text. The user must have delegated the scope the tool needs. The resource a call touches must belong to the run’s tenant, which the runtime verifies against its own records of ownership.
- Argument constraints bound the tool’s reach: a refund ceiling, an email recipient allowlist, a daily cap per tenant.
- Budgets bound the run: a maximum number of tool calls, so that a loop cannot continue indefinitely.
- Thresholds route consequential actions to a human. A refund above a limit is neither allowed nor denied; it requires approval.
Arguments are normalized before any rule runs. The demo converts a refund amount given as "30.00", 30, or 30.0 to the same integer number of cents. Without normalization, two equivalent requests would produce different decisions and different idempotency keys, and a rule that compares strings could be bypassed with formatting.
The code has four parts. The credential module follows the delegation semantics above.
"""Short-lived, audience-bound credentials for tool calls, with delegation semantics (user = sub, agent = act)."""
import base64, hashlib, hmac, json
SECRET = b"placeholder-signing-key-for-this-demo" # a real system keeps this in a key service and rotates it
class FakeClock:
def __init__(self): self.t = 1000.0
def __call__(self): return self.t
def advance(self, s): self.t += s
class AuthError(Exception): pass
def mint(clock, *, user, agent, tenant, audience, scopes, ttl=60):
body = json.dumps(dict(sub=user, act=agent, tenant=tenant, aud=audience, scope=sorted(scopes), exp=clock() + ttl), sort_keys=True)
sig = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()
return base64.b64encode(body.encode()).decode() + "." + sig
def verify(clock, token, *, audience, scope, tenant):
raw, sig = token.rsplit(".", 1)
body = base64.b64decode(raw)
if not hmac.compare_digest(sig, hmac.new(SECRET, body, hashlib.sha256).hexdigest()): raise AuthError("bad signature")
c = json.loads(body)
if c["exp"] <= clock(): raise AuthError("token expired")
if c["aud"] != audience: raise AuthError(f"token is for {c['aud']}, not {audience}")
if scope not in c["scope"]: raise AuthError(f"missing scope {scope}")
if c["tenant"] != tenant: raise AuthError("token tenant does not match resource tenant")
return cThe downstream services are fake. Payments verifies the token’s audience, scope, and tenant against the order’s owner, and it remembers results by idempotency key. Mail checks its token too, but has no idempotency support. Each can be told to commit a call and then lose the response, which is the failure that causes duplicates.
"""Two fake downstream services. Payments honors an idempotency key; Mail does not."""
from creds import verify
class Timeout(Exception): pass
class Payments:
audience = "payments"
def __init__(self, clock):
self.clock, self.refunds, self.by_key, self.owner = clock, [], {}, {"o-1001": "acme", "o-2002": "globex"}
self.lose_next_response = False
def refund(self, token, order_id, amount_cents, idempotency_key=None):
verify(self.clock, token, audience=self.audience, scope="refund:create", tenant=self.owner[order_id])
if idempotency_key and idempotency_key in self.by_key:
return self.by_key[idempotency_key] # replay the recorded result, do not refund again
result = {"refund_id": f"r-{len(self.refunds) + 1}", "order": order_id, "amount_cents": amount_cents}
self.refunds.append(result)
if idempotency_key: self.by_key[idempotency_key] = result
if self.lose_next_response:
self.lose_next_response = False
raise Timeout("response lost after the refund was committed")
return result
class Mail:
audience = "mail"
def __init__(self, clock): self.clock, self.sent, self.lose_next_response = clock, [], False
def send(self, token, tenant, to, subject):
verify(self.clock, token, audience=self.audience, scope="mail:send", tenant=tenant)
self.sent.append((to, subject))
if self.lose_next_response:
self.lose_next_response = False
raise Timeout("response lost after the message was sent")
return {"queued": True}The runtime holds the registry, normalization, policy, approvals, the hash-chained audit log, and the executor.
import hashlib, json
from dataclasses import dataclass, field
from creds import mint, AuthError
from services import Timeout
@dataclass
class Run:
run_id: str; tenant: str; user: str; agent: str
tools: set; scopes: set # tools the agent has, scopes the user delegated
max_steps: int = 10; steps_used: int = 0
steps: dict = field(default_factory=dict) # step_id -> {"status", "result", "key"}: the durable run state
# ---- tool registry: what each tool is, what it needs, how to recognize "the same action" ----------------
def cents(v): return round(float(v) * 100) if isinstance(v, str) or isinstance(v, float) else int(v) * 100
TOOLS = {
"lookup_order": dict(effect="read", scope="orders:read", dedupe=None, idempotent_downstream=True),
"issue_refund": dict(effect="write", scope="refund:create", dedupe="run", idempotent_downstream=True),
"send_email": dict(effect="send", scope="mail:send", dedupe="step", idempotent_downstream=False),
}
def normalize(tool, args):
a = dict(args)
if tool == "issue_refund": a["amount_cents"] = cents(a.pop("amount")) # "30.00", 30, and 30.0 mean the same thing
return a
def call_hash(run, tool, nargs): return hashlib.sha256(json.dumps([run.run_id, tool, nargs], sort_keys=True).encode()).hexdigest()[:16]
def idem_key(run, step_id, tool, nargs):
scope = TOOLS[tool]["dedupe"]
return hashlib.sha256(json.dumps([run.run_id, tool, nargs, step_id if scope == "step" else None], sort_keys=True).encode()).hexdigest()[:16]
# ---- policy engine: ordered rules, default deny, every decision names its rule ----------------------------
LIMITS = dict(refund_max=20000, refund_auto=5000, refund_daily_per_tenant=30000, mail_domains={"example.com"})
def decide(run, tool, nargs, state):
if tool not in run.tools: return "DENY", "R1-tool-not-granted"
if TOOLS[tool]["scope"] not in run.scopes: return "DENY", "R2-scope-not-delegated"
if run.steps_used >= run.max_steps: return "DENY", "R3-step-budget"
if "order_id" in nargs and state["owner"].get(nargs["order_id"]) != run.tenant: return "DENY", "R4-cross-tenant"
if tool == "issue_refund":
amt = nargs["amount_cents"]
if amt > LIMITS["refund_max"]: return "DENY", "R5-refund-over-max"
if sum(a for t, a in state["refunds"].values() if t == run.tenant) + amt > LIMITS["refund_daily_per_tenant"]: return "DENY", "R6-daily-cap"
if amt > LIMITS["refund_auto"]: return "REQUIRE_APPROVAL", "R7-refund-needs-approval"
if tool == "send_email" and nargs["to"].split("@")[-1] not in LIMITS["mail_domains"]: return "DENY", "R8-recipient-domain"
return "ALLOW", "default-allow-after-checks"
# ---- approvals bound to one exact call ---------------------------------------------------------------------
class Approvals:
def __init__(self, clock): self.clock, self.items = clock, {}
def request(self, run, tool, nargs):
aid = f"ap-{len(self.items) + 1}"
self.items[aid] = dict(call=call_hash(run, tool, nargs), requester=run.user, status="pending", expires=self.clock() + 900)
return aid
def approve(self, aid, approver):
it = self.items[aid]
if approver == it["requester"]: raise PermissionError("requester cannot approve their own request")
it.update(status="approved", approver=approver)
def consume(self, aid, run, tool, nargs):
it = self.items.get(aid)
if not it or it["status"] != "approved": return "no approved request"
if it["expires"] <= self.clock(): return "approval expired"
if it["call"] != call_hash(run, tool, nargs): return "approval was for a different call"
it["status"] = "used"; return None
# ---- hash-chained audit log -----------------------------------------------------------------------------------
class Audit:
def __init__(self): self.entries, self.prev = [], "0" * 16
def write(self, **fields):
e = dict(fields, prev=self.prev)
e["hash"] = hashlib.sha256(json.dumps(e, sort_keys=True).encode()).hexdigest()[:16]
self.entries.append(e); self.prev = e["hash"]
def verify(self):
prev = "0" * 16
for i, e in enumerate(self.entries):
body = {k: v for k, v in e.items() if k != "hash"}
if e["prev"] != prev or hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()[:16] != e["hash"]: return i
prev = e["hash"]
return None
# ---- executor ----------------------------------------------------------------------------------------------------------
class Executor:
def __init__(self, clock, payments, mail, approvals, audit, use_keys=True):
self.clock, self.pay, self.mail, self.ap, self.audit, self.use_keys = clock, payments, mail, approvals, audit, use_keys
self.state = dict(owner=payments.owner, refunds={}) # refunds: key -> (tenant, cents), for the daily cap
self.done_keys = {} # run-scope dedupe: key -> recorded result
self.crash_before_record = False
def _dispatch(self, run, tool, nargs, key):
if tool == "lookup_order":
return {"order": nargs["order_id"], "status": "shipped"} # a local read in this demo
aud = {"issue_refund": "payments", "send_email": "mail"}[tool]
token = mint(self.clock, user=run.user, agent=run.agent, tenant=run.tenant, audience=aud, scopes={TOOLS[tool]["scope"]}, ttl=60)
if tool == "issue_refund":
return self.pay.refund(token, nargs["order_id"], nargs["amount_cents"], key if self.use_keys else None)
if tool == "send_email":
return self.mail.send(token, run.tenant, nargs["to"], nargs["subject"])
def execute(self, run, step_id, tool, args, approval=None):
nargs = normalize(tool, args)
rec = run.steps.get(step_id)
if rec and rec["status"] == "done": # a replayed step returns its recorded result
return "replayed", rec["result"]
if rec and rec["status"] == "unknown":
return "needs-reconciliation", None
key = idem_key(run, step_id, tool, nargs) if tool in TOOLS and TOOLS[tool]["dedupe"] else None
if key in self.done_keys: # the same action, issued again under another step id
self.audit.write(run=run.run_id, step=step_id, tool=tool, decision="ALLOW", rule="dedupe", outcome="duplicate-suppressed", key=key)
return "duplicate-suppressed", self.done_keys[key]
decision, rule = decide(run, tool, nargs, self.state)
approved = bool(rec and rec.get("approved")) # an approval consumed before a crash still counts
if decision == "REQUIRE_APPROVAL":
problem = None if approved else (self.ap.consume(approval, run, tool, nargs) if approval else "no approval presented")
if problem:
self.audit.write(run=run.run_id, step=step_id, tool=tool, decision="HELD", rule=rule, detail=problem)
return "held", problem
approved = True; rule += "+approved"
elif decision == "DENY":
self.audit.write(run=run.run_id, step=step_id, tool=tool, decision="DENY", rule=rule)
return "denied", rule
run.steps_used += 1
run.steps[step_id] = dict(status="started", key=key, result=None, approved=approved)
try:
result = self._dispatch(run, tool, nargs, key)
except Timeout:
if TOOLS[tool]["idempotent_downstream"]: # safe to retry with the same key
result = self._dispatch(run, tool, nargs, key)
else: # outcome unknown: do not guess
run.steps[step_id]["status"] = "unknown"
self.audit.write(run=run.run_id, step=step_id, tool=tool, decision="ALLOW", rule=rule, outcome="unknown", key=key)
return "unknown", None
if self.crash_before_record:
self.crash_before_record = False
raise RuntimeError("runtime crashed after the side effect, before recording it")
if tool == "issue_refund": self.state["refunds"][key] = (run.tenant, nargs["amount_cents"])
if key and TOOLS[tool]["dedupe"] == "run": self.done_keys[key] = result
run.steps[step_id].update(status="done", result=result)
self.audit.write(run=run.run_id, step=step_id, tool=tool, decision="ALLOW", rule=rule, outcome="ok", key=key)
return "ok", resultThe demo exercises the pieces in seven scenarios. The keys and names in it are placeholders.
import json
from creds import FakeClock, mint, verify, AuthError
from services import Payments, Mail
from runtime import Run, Executor, Approvals, Audit, normalize
def world(use_keys=True):
clock = FakeClock(); pay, mail, audit = Payments(clock), Mail(clock), Audit()
return clock, pay, mail, audit, Executor(clock, pay, mail, Approvals(clock), audit, use_keys)
def new_run(rid="run-1", **kw):
base = dict(run_id=rid, tenant="acme", user="dana", agent="support-agent",
tools={"lookup_order", "issue_refund", "send_email"}, scopes={"orders:read", "refund:create", "mail:send"})
return Run(**{**base, **kw})
def show(label, out): print(f" {label:<46s} -> {out[0]}" + (f" ({out[1]})" if out[0] in ("denied", "held") else ""))
print("1. Policy decisions (default deny, every decision names its rule)")
clock, pay, mail, audit, ex = world(); run = new_run()
show("lookup_order o-1001", ex.execute(run, 1, "lookup_order", dict(order_id="o-1001")))
show("refund 30.00 on o-1001", ex.execute(run, 2, "issue_refund", dict(order_id="o-1001", amount="30.00")))
show("refund 120.00 on o-1001 (above auto limit)", ex.execute(run, 3, "issue_refund", dict(order_id="o-1001", amount=120)))
show("refund 250.00 on o-1001 (above max)", ex.execute(run, 4, "issue_refund", dict(order_id="o-1001", amount=250)))
show("refund 10.00 on o-2002 (other tenant's order)", ex.execute(run, 5, "issue_refund", dict(order_id="o-2002", amount=10)))
show("email to attacker@evil.test", ex.execute(run, 6, "send_email", dict(to="attacker@evil.test", subject="export")))
show("delete_account (tool not granted)", ex.execute(run, 7, "delete_account", dict(user="x")))
show("refund by run without refund scope", ex.execute(new_run("run-2", scopes={"orders:read"}), 1, "issue_refund", dict(order_id="o-1001", amount=5)))
print("\n2. Approval bound to one exact call")
call = dict(order_id="o-1001", amount=120)
aid = ex.ap.request(run, "issue_refund", normalize("issue_refund", call))
try: ex.ap.approve(aid, "dana")
except PermissionError as e: print(" requester approves own request ->", e)
ex.ap.approve(aid, "lee")
show("execute with a changed amount (150.00)", ex.execute(run, 8, "issue_refund", dict(order_id="o-1001", amount=150), approval=aid))
aid = ex.ap.request(run, "issue_refund", normalize("issue_refund", call)); ex.ap.approve(aid, "lee")
show("execute the approved call (120.00)", ex.execute(run, 3, "issue_refund", call, approval=aid))
print(f" approval status after use: {ex.ap.items[aid]['status']}; refunds made in this run: {sum(1 for r in pay.refunds if r['amount_cents'] == 12000)}")
print("\n3. Response lost after the refund committed")
for use_keys in (False, True):
clock, pay, mail, audit, ex = world(use_keys); run = new_run(); pay.lose_next_response = True
out = ex.execute(run, 1, "issue_refund", dict(order_id="o-1001", amount="30.00"))
made = len(pay.refunds)
again = ex.execute(run, 2, "issue_refund", dict(order_id="o-1001", amount=30)) # the model asks again, formatted differently
print(f" idempotency keys {'on ' if use_keys else 'off'}: first call {out[0]} with {made} refund(s) made; "
f"model repeats the call -> {again[0]}, {len(pay.refunds)} refund(s) in total")
print("\n4. A tool whose downstream cannot deduplicate (email)")
clock, pay, mail, audit, ex = world(); run = new_run(); mail.lose_next_response = True
out = ex.execute(run, 1, "send_email", dict(to="customer@example.com", subject="Your refund"))
print(f" send_email -> {out[0]}; messages sent: {len(mail.sent)}; step status: {run.steps[1]['status']}")
print(f" same step again -> {ex.execute(run, 1, 'send_email', dict(to='customer@example.com', subject='Your refund'))[0]}")
print("\n5. Crash after a side effect, then resume from the stored run state")
clock, pay, mail, audit, ex = world(); run = new_run()
plan = [("lookup_order", dict(order_id="o-1001")), ("issue_refund", dict(order_id="o-1001", amount="30.00")),
("send_email", dict(to="customer@example.com", subject="Your refund"))]
def run_plan(label):
try:
for i, (tool, args) in enumerate(plan, 1):
if label == "first attempt" and tool == "issue_refund": ex.crash_before_record = True
print(f" {label}: step {i} {tool:<13s} -> {ex.execute(run, i, tool, args)[0]}")
except RuntimeError as e:
print(f" {label}: crash: {e}")
run_plan("first attempt"); run_plan("resume")
print(f" refunds made: {len(pay.refunds)}; emails sent: {len(mail.sent)}")
print("\n6. Credentials")
clock = FakeClock()
tok = mint(clock, user="dana", agent="support-agent", tenant="acme", audience="payments", scopes={"refund:create"}, ttl=60)
for label, kw in (("fresh token at payments", {}), ("same token used at mail", dict(audience="mail", scope="mail:send")),
("token for another tenant's order", dict(tenant="globex"))):
args = dict(audience="payments", scope="refund:create", tenant="acme") | kw
try: verify(clock, tok, **args); print(f" {label:<36s} -> accepted")
except AuthError as e: print(f" {label:<36s} -> rejected: {e}")
clock.advance(61)
try: verify(clock, tok, audience="payments", scope="refund:create", tenant="acme")
except AuthError as e: print(f" {'after 61 seconds':<36s} -> rejected: {e}")
print("\n7. Audit chain")
clock, pay, mail, audit, ex = world(); run = new_run()
ex.execute(run, 1, "issue_refund", dict(order_id="o-1001", amount="30.00"))
ex.execute(run, 2, "issue_refund", dict(order_id="o-2002", amount=10))
ex.execute(run, 3, "send_email", dict(to="customer@example.com", subject="Your refund"))
for e in audit.entries: print(" ", json.dumps(e))
print(" chain intact:", audit.verify() is None)
audit.entries[1]["rule"] = "default-allow-after-checks"; audit.entries[1]["decision"] = "ALLOW"
print(" after editing entry 1, first bad entry:", audit.verify())1. Policy decisions (default deny, every decision names its rule)
lookup_order o-1001 -> ok
refund 30.00 on o-1001 -> ok
refund 120.00 on o-1001 (above auto limit) -> held (no approval presented)
refund 250.00 on o-1001 (above max) -> denied (R5-refund-over-max)
refund 10.00 on o-2002 (other tenant's order) -> denied (R4-cross-tenant)
email to attacker@evil.test -> denied (R8-recipient-domain)
delete_account (tool not granted) -> denied (R1-tool-not-granted)
refund by run without refund scope -> denied (R2-scope-not-delegated)
2. Approval bound to one exact call
requester approves own request -> requester cannot approve their own request
execute with a changed amount (150.00) -> held (approval was for a different call)
execute the approved call (120.00) -> ok
approval status after use: used; refunds made in this run: 1
3. Response lost after the refund committed
idempotency keys off: first call ok with 2 refund(s) made; model repeats the call -> duplicate-suppressed, 2 refund(s) in total
idempotency keys on : first call ok with 1 refund(s) made; model repeats the call -> duplicate-suppressed, 1 refund(s) in total
4. A tool whose downstream cannot deduplicate (email)
send_email -> unknown; messages sent: 1; step status: unknown
same step again -> needs-reconciliation
5. Crash after a side effect, then resume from the stored run state
first attempt: step 1 lookup_order -> ok
first attempt: crash: runtime crashed after the side effect, before recording it
resume: step 1 lookup_order -> replayed
resume: step 2 issue_refund -> ok
resume: step 3 send_email -> ok
refunds made: 1; emails sent: 1
6. Credentials
fresh token at payments -> accepted
same token used at mail -> rejected: token is for payments, not mail
token for another tenant's order -> rejected: token tenant does not match resource tenant
after 61 seconds -> rejected: token expired
7. Audit chain
{"run": "run-1", "step": 1, "tool": "issue_refund", "decision": "ALLOW", "rule": "default-allow-after-checks", "outcome": "ok", "key": "e71f75badc4f5804", "prev": "0000000000000000", "hash": "ca5585030445c9e7"}
{"run": "run-1", "step": 2, "tool": "issue_refund", "decision": "DENY", "rule": "R4-cross-tenant", "prev": "ca5585030445c9e7", "hash": "6c643757a38d8b30"}
{"run": "run-1", "step": 3, "tool": "send_email", "decision": "ALLOW", "rule": "default-allow-after-checks", "outcome": "ok", "key": "d9ec6e3d6cbc8e49", "prev": "6c643757a38d8b30", "hash": "b6a22f557c33ee94"}
chain intact: True
after editing entry 1, first bad entry: 1Reading the policy decisions
The first scenario shows the decisions. The lookup and the 30-dollar refund run. The 120-dollar refund is above the automatic limit, so it is held, which means it is neither executed nor refused permanently. The 250-dollar refund is above the maximum and is denied by rule R5. The refund on an order owned by another tenant is denied by R4 before any credential is minted, because the runtime checks the order’s owner against the run’s tenant. The payments service would also reject the call, since the token’s tenant does not match the order’s owner. The two checks are independent, and the first gives a precise, audited reason. The email to an address outside the allowed domain is denied by R8, which is the exfiltration channel from the opening story. A tool that was never granted is denied by R1, and a run whose user did not delegate the refund scope is denied by R2. Each denial names its rule, and the audit entry records it. None of these decisions asks the model anything.
Two design points stand out. Order matters: the checks run from the cheapest and broadest (is this tool granted at all) to the most specific (is this amount allowed), and the first failure decides. And policy lives in data and small functions that can be tested without a model. A platform should keep a table of cases, the inputs and the expected decision and rule, and run it on every change to the policy, since a policy bug is a security bug.
Approval bound to one call
A policy that returns REQUIRE_APPROVAL creates an obligation that a person must meet before the call runs. Approval can fail in ways that approving “the agent’s refund” does not: the approver might see a summary the model wrote, the approved action might differ from the one executed, or the approval might be reused.
The demo ties an approval to a hash of the run, the tool, and the normalized arguments. The scenario shows the consequences. The user who started the run cannot approve their own request, which is separation of duties. An approval for a 120-dollar refund cannot be used for a 150-dollar one, because the hashes differ, so the call is held again with the reason. The approved call then executes, and the approval’s status becomes used, so it cannot be spent twice. Approvals also expire (15 minutes in the demo). A change in the arguments between approval and execution is a time-of-check to time-of-use gap, and binding the approval to the call’s hash closes it.
The approver must see what will actually happen. The approval screen should render the tool name and the normalized arguments from the runtime, not a description produced by the model, since a description is text the model can be steered into writing. MCP’s specification makes the same recommendation for clients: tell the user which tools are exposed, show when they are invoked, and present confirmation prompts for operations. OWASP adds that approval can live in the downstream system as well as in the extension itself. High-impact actions that cannot be undone, such as sending money or messages to outsiders, are the ones that most need it, and a threshold that is too low trains approvers to click through, which defeats the control.
Run state, memory, and resume
An agent run can last minutes or hours, calls slow services, and can be interrupted by a deployment, a crash, or a timeout. The runtime therefore keeps the state of a run outside the process that executes it. In the demo, Run.steps records for each step an identifier, a status (started, done, or unknown), the idempotency key, and the result. With that record, a new process can pick up the run: finished steps return their recorded results without executing again, steps that were started but not finished are re-attempted, and steps whose outcome is unknown are flagged. In production this record lives in a durable store, scoped by tenant, and it is written before the side effect (the started mark) and after it. The conversation context given to the model can be rebuilt from it.
Long-term memory is a different thing from run state. It is information that persists across runs, such as a customer’s preferences or a summary of earlier interactions. It needs its own scoping, by tenant and by user, its own retention and deletion, and a write policy, because a memory that an attacker can write to through a poisoned document becomes a persistent channel for instructions. The runtime should treat memory writes as actions subject to the same policy and audit as any other tool call.
Idempotency: not doing it twice
Duplicates in an agent system come from several places: a call times out and the runtime retries, a worker crashes between performing an action and recording that it did, the model sees an error and issues the same call again, and an orchestrator replays a run. In each case the cause is the same: the caller does not know whether the first attempt happened. Retrying is correct only if repeating the action has the same effect as doing it once. Idempotency states the property and the usual arrangement: at-least-once delivery combined with idempotent application, in which a key identifies the logical operation so the destination can tell a retry from a new request.
Stripe’s API documentation describes a widely used implementation of the client-supplied key: the server saves the status code and body of the first request made with a given key and returns the same result for later requests with that key, including a 500; it compares incoming parameters with the original request and errors if they differ; keys are up to 255 characters; and keys can be removed once they are at least 24 hours old. Two details matter for agents. The comparison of parameters turns accidental key reuse into an error and not a silent success. And the retention window means retries after a long delay need a plan.
The central design question is what counts as the same action, because the key encodes the answer. The demo registry uses two policies.
- Run scope. The key is a hash of the run, the tool, and the normalized arguments, and excludes the step. “Refund 30.00 on order o-1001 in this run” is one action however many times it is requested. This is right for operations with a natural business identity, and it suppresses the model asking again.
- Step scope. The key includes the step identifier, so each step is a distinct action even with identical arguments. This is right for operations that may legitimately repeat, such as sending the same reminder on two different days in two steps. A model that re-asks after a lost response creates a new step and defeats this protection, which is why step scope is appropriate only where the runtime, not the model, controls retries.
The third scenario shows the effect of the key. The payments service commits the refund and loses the response. Without a key, the runtime’s retry creates a second refund, and the output shows two refunds. With a key, the retry reaches the service with the same key, which replays the recorded result: one refund. The model then repeats the call, formatted differently ("30.00" against 30), and normalization makes the two requests identical, so the runtime recognizes the duplicate and does not call the service at all. Both layers matter. The downstream key protects against retries in the network path. The runtime-level check protects against the model’s behavior and against downstream services that lack keys.
Some tools cannot be made idempotent by a key. Mail in the demo has no deduplication, and the fourth scenario shows the correct behavior when its response is lost: the runtime does not retry. It marks the step unknown and returns a state that requires reconciliation, for example by looking up whether the message was sent, since an automatic retry risks sending twice and an automatic skip risks sending nothing. Choosing at-most-once for such tools is a decision about which error is cheaper. An outbox, in which the runtime writes the intent durably and a separate sender delivers it with its own deduplication, is the usual route to a better guarantee.
The fifth scenario combines this with run state. The runtime crashes after the refund is committed but before recording it. On resume, the lookup step returns its recorded result without executing, the refund step is re-attempted with the same key and the payments service replays the result (one refund in total), and the email step executes once. The approval consumed before the crash is also remembered in the step record, so a resumed step does not demand a second approval. The key was derived from the run and the arguments, which are stable across restarts, and not from anything that changes with a process, such as a counter in memory.
Audit of actions
The audit log answers what the agent did, why it was allowed, who approved it, and what happened. Audit Log notes that a request and its outcome are different events and that the record has to carry identifiers that let them be correlated. The demo writes one entry per decision with the run, step, tool, decision, rule, outcome, and idempotency key, plus the approver for approved calls, and chains the entries: each contains the hash of the previous one. The seventh scenario shows the check. The chain verifies, then an edit to entry 1 makes verification fail at that entry.
A hash chain makes undetected edits harder, not impossible. Someone with write access to the whole log can rewrite every later hash. Tamper evidence needs the latest hash to be anchored somewhere the log writer cannot change, such as a separate store or a periodic signature. The log records arguments as hashes or selected fields, not in full, because arguments can contain personal data, and the retention and access rules for it belong to the same discussion as the platform’s other logs. For actions with external effects, the log should also capture the downstream identifier of what was created, such as the refund’s identifier, so the action can be located and, where possible, compensated.
Acceptance checks and failure modes
The demo’s scenarios double as acceptance checks for a real runtime. Before a class of agent is allowed to run for tenants, the platform should be able to show that:
- A call to a tool outside the agent’s grant, or needing an undelegated scope, is refused with a named rule.
- A call that touches another tenant’s resource is refused even when the credential used would be accepted downstream.
- An action above a threshold waits for an approval, and an approval cannot be used for a different call, reused, or approved by the requester.
- A lost response followed by a retry produces one effect, and a re-issued, differently formatted request produces no second effect.
- A tool without downstream deduplication ends in an unknown state and not a blind retry.
- A crash between a side effect and its record does not lose or repeat the effect after resume.
- A token expires, is rejected at another audience, and is rejected for another tenant.
- An edit to the audit log is detected.
- The isolation tier for code-running tools has been tested for the things it is supposed to block, the way the coding-agent article tests denied file access, denied network destinations, and resource limits, and that allowed work still succeeds.
The runtime has its own failure modes. If the policy engine is unreachable, the choice between failing open and failing closed has to be made in advance: for actions with external effects, closed is the safe default. A policy that is too strict drives teams to bypass the platform. A policy that is too loose gives the model’s mistakes real consequences. Approval fatigue turns a control into a ritual when the threshold generates too many requests. Unknown steps that nobody reconciles accumulate. A step budget that is too generous allows a loop to run up cost before it stops, and one that is too tight interrupts legitimate work. And every one of these controls reduces risk without removing it: prompt injection still shapes what the model proposes, and the runtime’s job is to make sure that what it proposes can do only bounded, recorded, reversible or approved harm.
What comes next
The runtime produces decisions, actions, approvals, and audit entries, and the serving layer and gateway produce latency, errors, and usage. The next module of this course turns those into operational signals. Observing AI Systems Without Exposing Sensitive Data covers how to trace and monitor prompts, tool calls, and outputs while keeping personal and confidential data out of the logs that engineers read.
References
- OWASP GenAI Security Project, LLM06:2025 Excessive Agency
- Model Context Protocol specification (latest)
- Model Context Protocol specification, Tools
- Model Context Protocol specification, Authorization
- Model Context Protocol specification, Security Best Practices
- RFC 8693, OAuth 2.0 Token Exchange
- Stripe API reference, Idempotent requests
- Firecracker microVM project
- gVisor documentation, What is gVisor?
- Kubernetes documentation, Runtime Class
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.