Controlling AI Cost per Successful Task
Consider a hypothetical platform team whose assistant answers internal support tickets. The monthly model bill is growing, so the team routes all traffic to a smaller, cheaper model. The cost dashboard, which divides spend by request count, drops by about 80 percent within a day. Two weeks later the support queue holds more reopened tickets, and a reviewer notices that people are correcting the assistant’s drafts by hand. The dashboard still says the change was a success, because the number it reports is cost per request, and the requests got cheaper.
The quantity the business pays for is different. A ticket that the assistant handles correctly is worth the model spend. A ticket that it handles incorrectly costs the model spend and then a person’s time, and sometimes a customer’s patience. A cost metric that ignores the second case rewards the cheapest wrong answer. This article defines the metric that does not, cost per successful task, and works through how each cost component enters it, how retries and agent loops multiply it, and how caching and model routing change it. It then runs a local simulation that includes a policy whose cost per request is nearly seven times lower than the baseline while its cost per successful task is almost five times higher.
The article assumes you have read AI Platform Security and Tenant Isolation, in particular the idea that a shared platform keeps one tenant’s data and credentials away from another’s. That matters here because caching, the biggest cost lever, is also a way for one tenant’s data to reach another. It also assumes that requests pass through a model gateway that can attribute usage to a tenant and application, and that evaluation results are versioned, as in the earlier articles on model gateways and evaluation gates. General programming and probability at the level of expected values are enough for the rest.
Prices in this article are assumed unit prices. They were read from the Anthropic API pricing page and the prompt caching documentation in October 2026, and they are inputs to a calculation, not a recommendation of a vendor. Every dollar figure below is computed from those assumptions and from a synthetic workload. Replace the constants with your own and the method still applies.
From cost per request to cost per successful task
A request is one call to a model. A task is one unit of work that a user or a system wanted done: answer this ticket, fix this failing test, extract these fields from this document. A task can take one request, several requests when a program retries, or dozens when an agent loops through tool calls. Billing systems see requests, because a request is what a provider meters. Users see tasks.
The metric this article uses is the total cost of handling a set of tasks, divided by the number of tasks whose delivered result was correct.
\[ \text{CPST} = \frac{\sum_{t \in \text{tasks}} \left( C^{\text{gen}}_t + C^{\text{check}}_t + C^{\text{human}}_t + C^{\text{fail}}_t \right)}{\#\{t : \text{delivered result of } t \text{ is correct}\}} \]
The numerator sums four costs over every task, including the ones that failed. \(C^{\text{gen}}\) is the spend on generation: tokens, or GPU time for self-hosted models, across every attempt. \(C^{\text{check}}\) is the spend on deciding whether an output is acceptable, such as an automated test, a schema check, or a judge model. \(C^{\text{human}}\) is the cost of a person completing or fixing a task that the system could not. \(C^{\text{fail}}\) is the cost of a wrong result that nobody caught before delivery: rework, a support contact, a refund. The denominator counts correct deliveries, which includes tasks a human fixed. Spending on failures is therefore charged to the successes, and a policy that wastes money on failed attempts pays for it in the ratio.
The ratio has two properties that a per-request figure lacks. First, it falls when success rises at constant spend, and rises when spend rises at constant success, so it responds to both levers a platform controls. Second, it is the right quantity to compare across configurations that make different numbers of requests per task. A cascade that makes three cheap calls and a single expensive call are comparable only per task.
Two measurement problems come with it. The denominator requires a notion of “correct”, which for most tasks is an estimate rather than a fact. The numerator requires that every request carry a task identifier, so retries and agent steps can be summed back to the task that caused them. Both are covered below. The first step is to see what goes into each cost term.
Where the money goes
Tokens
Hosted model APIs bill by tokens, and the price differs by direction and by cache state. A token is the unit of text the model reads and writes; a word is typically one to two tokens in English. Input tokens are everything in the prompt: the system prompt, tool definitions, retrieved documents, conversation history, and the task itself. Output tokens are what the model generates, and on these price lists they cost five times as much per token as input. The three models used as stand-ins in this article have the following list prices.
| Stand-in tier | Model | Input | Output | 5-minute cache write | Cache read | Minimum cacheable prompt |
|---|---|---|---|---|---|---|
| S (small) | Claude Haiku 4.5 | \$1 / MTok | \$5 / MTok | \$1.25 / MTok | \$0.10 / MTok | 4,096 tokens |
| M (mid) | Claude Sonnet 5.5 | \$2 / MTok | \$10 / MTok | \$2.50 / MTok | \$0.20 / MTok | 512 tokens |
| L (large) | Claude Opus 5.5 | \$4 / MTok | \$20 / MTok | \$5 / MTok | \$0.20 / MTok | 512 tokens |
MTok means one million tokens. The cache columns follow from multipliers on the input price: a five-minute cache write costs 1.25 times the input price and a cache read costs 0.1 times, except that Opus 5.5 reads at 0.05 times. One-hour cache writes cost 2 times the input price. Batch processing, which runs requests asynchronously, takes 50 percent off both input and output tokens and stacks with caching. These are the facts the rest of the cost model builds on, and any of them can change, which is why the calculations below keep prices in a single table of constants.
Two details in that table affect comparisons more than the headline prices do. The first is the tokenizer. The pricing page states that models from Claude 4.7 onward use a newer tokenizer that produces roughly 30 percent more tokens for the same text, while Haiku 4.5 uses the older one. A price per million tokens is therefore not a price per document across those models. The simulation below counts 1.3 times as many tokens for M and L as for S to reflect this. The second is the minimum cacheable length. A prompt prefix shorter than the minimum is simply processed without caching and returns no error, so a 3,000-token system prompt caches on M and L but silently does not cache on S.
Tool use adds tokens that are easy to forget. Tool definitions are part of the input, and the pricing page lists a fixed system-prompt overhead whenever a tool is present. Tool results are fed back as input on the next step. A server-side tool such as web search is billed in addition to tokens, at a stated price per thousand searches. For an agent, these tokens, not the user’s question, are often the bulk of the bill.
GPU time
When a team hosts a model itself, the invoice is a rate per GPU-hour, not a price per token, and the effective price per token depends on how busy the GPU is. If a node costs a fixed amount per hour and can generate a certain number of tokens per second when fully loaded, then the cost per million output tokens is that hourly rate divided by the tokens produced per hour at the actual utilization.
\[ \text{cost per MTok} = \frac{r}{v \cdot 3600 \cdot u} \times 10^{6} \]
Here \(r\) is the hourly rate in dollars, \(v\) is aggregate throughput in tokens per second at full load, and \(u\) is the fraction of time the capacity is producing useful tokens. The factor \(u\) is why scheduling matters for cost: capacity sitting idle between bursts of traffic is paid for but sells nothing. The article on GPU scheduling and isolation covers how to raise utilization. With hypothetical figures of 4 dollars per hour and 2,500 tokens per second (assumptions, not measurements of any hardware), the formula gives these costs.
self-hosted GPU: $ per million output tokens = rate / (tokens_per_s * 3600 * utilization) * 1e6
utilization 1.00: $ 0.444 per MTok
utilization 0.60: $ 0.741 per MTok
utilization 0.30: $ 1.481 per MTok
utilization 0.10: $ 4.444 per MTok
utilization 0.05: $ 8.889 per MTok
utilization needed to match an API output price of $5.0/MTok: 0.089
utilization needed to match an API output price of $10.0/MTok: 0.044At full utilization the self-hosted token is cheap, under half a dollar per million. At 5 percent utilization it costs more than the 5 dollar small-tier API output price and approaches the 10 dollar mid-tier price. The last two lines read the same formula backward: this hypothetical node must stay busy about 9 percent of the time to match a 5 dollar API price and about 4 percent to match a 10 dollar price. The numbers are artificial, but the shape is general. A self-hosted deployment is cheaper per token only when traffic keeps it busy, and idle capacity belongs in the numerator of cost per successful task like any other generation cost.
Evaluation
Evaluation costs money in two places. Offline evaluation runs a candidate model or prompt against a fixed dataset before release, as in the release gates of the earlier article. Online evaluation checks live outputs, for example with an automated test or a judge model, and its cost is paid on every task it covers. Offline cost is lumpy and tied to release frequency. Online cost scales with traffic, which makes it part of the per-task cost.
The offline cost is small enough to underestimate. Using the same synthetic workload as the simulation below, one pass over 1,500 evaluation cases on the large tier, each followed by a judge call on the small tier, comes to a few dozen dollars. Three candidate configurations over five random seeds each multiply that by fifteen. Because evaluation runs rarely need an immediate answer, the asynchronous batch interface, which the provider documents for large-scale evaluations at half the token price, is a direct lever here. The same page of the provider’s data retention documentation lists batch processing as not eligible for zero data retention, so confirm the data-handling terms before moving evaluation data that contains customer content to that interface.
evaluation run: 1,500 cases, generation on large tier plus judge on small tier
one pass: $44.12; 3 candidate configs x 5 seeds: $661.83That is a reasonable price for a release gate. It becomes a recurring line item when every prompt edit triggers the full set, so the number of evaluation runs per week belongs in the cost model next to the number of releases.
Retries and failures
A retry is a second generation for the same task, so it adds to \(C^{\text{gen}}\) without adding to the denominator unless it turns a failure into a success. Retries come from several sources: a provider error, a timeout, a failed schema validation, a failed test, or a judge that rejected the output. A platform that retries on every transient error at the gateway and also retries on every validation failure in the application can multiply requests per task by more than either layer intends. The only way to see this is to count requests per task, not requests per second.
Human review
A person enters the picture in two ways. In a human-in-the-loop design, a reviewer approves or edits outputs before they take effect. In a human-fallback design, the system tries first and escalates to a person when checks fail. Either way the cost is the reviewer’s time multiplied by a loaded hourly rate. This simulation assumes 5 minutes at 48 dollars per hour, or 4 dollars, per escalated task. Compare that with a generation cost of a few cents. With realistic labor rates, a human touch costs one to two orders of magnitude more than a model call, so the number of tasks that reach a person usually decides whether the unit economics work, regardless of token prices.
Defining success
The denominator of the metric is only as good as the definition of “correct”. Three definitions are common, and each has a failure mode.
- Executable checks. The output passes a test suite, a schema validator, or a business rule. These are cheap and exact for what they check. They miss anything not encoded in the check, so a patch that passes the tests can still be the wrong patch.
- Judge models. A second model grades the first against a rubric. A judge scales to open-ended outputs but has its own error rates, and its errors are not random: it tends to prefer outputs that resemble its own style.
- Human labels. A sampled audit by reviewers is the reference for the other two. It is expensive, so it covers a small fraction of tasks.
A checker has two error rates. The false-accept rate is the fraction of wrong outputs that it passes. The false-reject rate is the fraction of correct outputs that it blocks. They affect cost differently. A false reject sends a good task into a retry or to a person, which adds cost without changing the outcome. A false accept delivers a wrong result and charges \(C^{\text{fail}}\) later. The success rate the platform measures through the checker is also biased: with true success rate \(s\), false-accept rate \(a\), and false-reject rate \(b\), the checker reports an acceptance rate of \(s(1-b) + (1-s)a\). With the rates used below, a true success rate of 0.90 shows up as 0.883, and a true success rate of 0.60 shows up as 0.622. Operators who compare configurations on the checker’s own numbers will see the better configuration look somewhat worse and the worse one somewhat better. A periodic human audit of accepted outputs is what keeps the estimate honest.
The second parameter is the cost of a wrong result that reaches the user, \(R\) in the simulation. It is the hardest number to set and the one the answer is most sensitive to. A wrong autocomplete suggestion costs the user a second. A wrong refund decision costs money and a complaint. The tradeoff section below shows that \(R\) alone decides whether the cheap model or the expensive one is cheaper per successful task.
A simulation of cost per successful task
What is modeled
The experiment below is a local simulation, not a measurement of any provider. It uses a synthetic workload and invented success probabilities, and it uses the price table above only for unit costs. What it can show is how the accounting behaves under stated assumptions: which terms dominate, and which orderings of policies are robust to the assumptions. It cannot show how any real model performs. It was run with Python 3.12.14 on macOS (Darwin 25.6.0), using only the standard library.
The synthetic workload has three difficulty classes with weights 55 percent easy, 30 percent medium, and 15 percent hard. Every request carries a 3,000-token shared prefix (system prompt and tool schema), a task-specific input of 400, 900, or 1,800 tokens, and an output of 150, 350, or 800 tokens by class. Each tier succeeds with a probability that depends on the class: the small tier is good on easy tasks and poor on hard ones, the large tier is good on all three. A failed attempt retried on the same tier succeeds with only 0.6 of the original probability, because a task that a tier failed is more likely to be a hard instance for that tier than an average one in its class. The checker is a small-tier judge call that accepts 10 percent of wrong outputs and rejects 3 percent of correct ones. Tasks that exhaust the policy’s attempts go to a person at 4 dollars, who always succeeds. The constants look like this.
# ---- assumed unit prices (USD per million tokens), from the vendor price page ----
# name: (input, output, cache_write_5m, cache_read, min_cacheable_tokens, token_multiplier)
TIERS = {
"S": dict(inp=1.0, out=5.0, cw=1.25, cr=0.10, min_cache=4096, tok=1.0), # small tier
"M": dict(inp=2.0, out=10.0, cw=2.50, cr=0.20, min_cache=512, tok=1.3), # mid tier
"L": dict(inp=4.0, out=20.0, cw=5.00, cr=0.20, min_cache=512, tok=1.3), # large tier
}
# ---- assumed workload (tokens counted with the small tier's tokenizer) ----
PREFIX = 3000 # shared system prompt + tool schema
TASK_IN = {"easy": 400, "med": 900, "hard": 1800}
TASK_OUT = {"easy": 150, "med": 350, "hard": 800}
MIX = {"easy": 0.55, "med": 0.30, "hard": 0.15}
P = { # P(correct | tier, difficulty)
"S": {"easy": 0.93, "med": 0.55, "hard": 0.12},
"M": {"easy": 0.98, "med": 0.88, "hard": 0.55},
"L": {"easy": 0.99, "med": 0.96, "hard": 0.82},
}
FRR, FAR = 0.03, 0.10 # checker rejects a good answer / accepts a bad one
CHECK_EXTRA_IN, CHECK_OUT = 600, 40
H = 4.00 # human fix: ~5 min at an assumed $48/h loaded rate
RETRY_DECAY = 0.6 # a failed instance is harder than average for that tier
CACHE_HIT = 0.95 # share of requests whose prefix is already cached
SEM_HIT, SEM_RIGHT = 0.25, 0.85Cost per call applies the cache rules. When prompt caching is on and the prefix meets the tier’s minimum length, 95 percent of requests read the prefix from cache and 5 percent write it. Otherwise the prefix is billed at the input price. Output tokens are never cached.
def call_cost(tier, task_in, out, cache):
t = TIERS[tier]; m = t["tok"]
pre, ti, o = PREFIX * m, task_in * m, out * m
if cache and pre >= t["min_cache"]:
pre_cost = pre * (CACHE_HIT * t["cr"] + (1 - CACHE_HIT) * t["cw"])
else:
pre_cost = pre * t["inp"]
return (pre_cost + ti * t["inp"] + o * t["out"]) / 1e6
def check_cost(task_out_tokens, task_in):
t = TIERS["S"]
return ((task_in + task_out_tokens + CHECK_EXTRA_IN) * t["inp"] + CHECK_OUT * t["out"]) / 1e6The simulation function walks each task through its policy. A policy is a list of tiers to try in order, a flag for whether to check each output, and options for prompt caching, a semantic cache, and a router. The core loop is below. Each attempt costs a generation call and, when checking is on, a judge call. An accepted output is delivered, correct or not. A task that runs out of attempts goes to a person. Without a checker, the first output is delivered as is.
def run(pol, n, R, seed=7):
r = random.Random(seed)
api = human = rework = chk = 0.0
reqs = ok = esc = hum = 0
for _ in range(n):
d = pick_difficulty(r)
if pol.sem and r.random() < SEM_HIT:
if r.random() < SEM_RIGHT: ok += 1
else: rework += R; esc += 1
continue
chain = pol.chain
if pol.router:
pred = d if r.random() < 0.8 else r.choice([x for x in MIX if x != d])
start = {"easy": "S", "med": "M", "hard": "L"}[pred]
order = ["S", "M", "L"]
chain = order[order.index(start):]
api += call_cost("S", 300, 5, False); reqs += 1
seen = {}
done = False
for tier in chain:
k = seen.get(tier, 0); seen[tier] = k + 1
p = P[tier][d] * (RETRY_DECAY if k else 1.0)
api += call_cost(tier, TASK_IN[d], TASK_OUT[d], pol.cache); reqs += 1
good = r.random() < p
if not pol.check:
if good: ok += 1
else: rework += R; esc += 1
done = True; break
c = check_cost(TASK_OUT[d], TASK_IN[d]); api += c; chk += c; reqs += 1
accepted = (r.random() > FRR) if good else (r.random() < FAR)
if accepted:
if good: ok += 1
else: rework += R; esc += 1
done = True; break
if not done:
human += H; hum += 1; ok += 1
total = api + human + rework
return dict(name=pol.name, n=n, succ=ok / n, esc=esc / n, hum=hum / n,
req_per_task=reqs / n, per_req=api / reqs if reqs else 0,
per_task=total / n, per_succ=total / ok, api=api / n,
chk=chk / n, human_cost=human / n, rework_cost=rework / n)Nine policies are compared. Three use one tier and no checking. One uses the large tier with a checker. One retries the small tier twice, three attempts in all, with a checker. One cascades small, mid, then large with a checker, and two variants add prompt caching and then a semantic cache that serves 25 percent of tasks from stored answers, 85 percent of which are correct. The last uses a router that predicts task difficulty correctly 80 percent of the time and starts at the matching tier, escalating on rejection. The router’s own classification call is billed at small-tier prices.
Results with no cost for undetected failures
First, set \(R = 0\), so a wrong answer that nobody catches costs nothing downstream. This is how a cost dashboard implicitly treats the world. The table lists success rate, the fraction of wrong deliveries (esc), the fraction sent to a person (hum), requests per task, dollars per request, dollars per task, dollars per successful task, and a star for policies not beaten on both cost per task and success rate.
R (cost of one delivered wrong answer) = $0.00, H = $4.00, N = 200000
policy succ esc hum req/t $/req $/task $/succ front
L only, no check 0.955 0.045 0.000 1.00 0.0275 0.0275 0.0288 *
M only, no check 0.885 0.115 0.000 1.00 0.0138 0.0138 0.0156 *
S only, no check 0.695 0.305 0.000 1.00 0.0053 0.0053 0.0076 *
L + check 0.996 0.004 0.069 2.00 0.0147 0.3041 0.3055 *
S x3 retries + check 0.932 0.068 0.140 2.99 0.0040 0.5715 0.6132
S>M>L cascade + check 0.961 0.039 0.010 2.73 0.0059 0.0578 0.0602
cascade + prompt cache 0.961 0.039 0.010 2.73 0.0048 0.0549 0.0571 *
cascade + cache + semantic cache 0.934 0.066 0.008 2.04 0.0048 0.0415 0.0445
router + escalate + cache 0.987 0.013 0.031 3.23 0.0050 0.1420 0.1439 *The cheapest policy per successful task is the small tier alone: 0.0076 dollars per successful task against 0.0288 for the large tier. Policies that add a checker become much more expensive per success, 0.0571 for the cascade and 0.3055 for the large tier with a checker. The reason is in the hum column. With a single tier there is nowhere to escalate except a person, so every output the checker rejects, whether the output was good (3 percent) or wrong and caught (90 percent of the wrong ones), goes to a person, and each such task costs 4 dollars. With the large tier and a checker, 6.9 percent of tasks go to a person, which is 0.27 dollars per task, ten times the entire generation cost. When undetected wrong answers cost nothing, verification is pure expense, and the cheapest strategy is to ship the first output.
Results when a wrong answer costs 3 dollars
Now set \(R = 3\), meaning each wrong result that reaches a user costs 3 dollars in rework or support.
R (cost of one delivered wrong answer) = $3.00, H = $4.00, N = 200000
policy succ esc hum req/t $/req $/task $/succ front
L only, no check 0.955 0.045 0.000 1.00 0.0275 0.1614 0.1690 *
M only, no check 0.885 0.115 0.000 1.00 0.0138 0.3574 0.4037
S only, no check 0.695 0.305 0.000 1.00 0.0053 0.9189 1.3212
L + check 0.996 0.004 0.069 2.00 0.0147 0.3176 0.3190 *
S x3 retries + check 0.932 0.068 0.140 2.99 0.0040 0.7757 0.8323
S>M>L cascade + check 0.961 0.039 0.010 2.73 0.0059 0.1758 0.1830
cascade + prompt cache 0.961 0.039 0.010 2.73 0.0048 0.1729 0.1800 *
cascade + cache + semantic cache 0.934 0.066 0.008 2.04 0.0048 0.2401 0.2571
router + escalate + cache 0.987 0.013 0.031 3.23 0.0050 0.1798 0.1821 *The ordering reverses. The small tier alone is now the most expensive per success at 1.3212 dollars, because 30.5 percent of its deliveries are wrong. The large tier alone costs 0.1690, and the cascade with prompt caching costs 0.1800. The star column marks the large tier alone, the large tier with a checker, the cascade with prompt caching, and the router as the policies not beaten on both cost per task and success rate. They span a real tradeoff: the large tier with a checker reaches the highest success rate, 0.996, at nearly twice the cost per task of the large tier alone.
The breakdown of cost per task at this setting shows what the totals are made of.
cost per task, R = $3.00
L only, no check gen 0.0275 check 0.0000 human 0.0000 rework 0.1339 total 0.1614
M only, no check gen 0.0138 check 0.0000 human 0.0000 rework 0.3437 total 0.3574
S only, no check gen 0.0053 check 0.0000 human 0.0000 rework 0.9136 total 0.9189
L + check gen 0.0276 check 0.0019 human 0.2747 rework 0.0134 total 0.3176
S x3 retries + check gen 0.0087 check 0.0031 human 0.5597 rework 0.2042 total 0.7757
S>M>L cascade + check gen 0.0133 check 0.0028 human 0.0417 rework 0.1180 total 0.1758
cascade + prompt cache gen 0.0104 check 0.0028 human 0.0417 rework 0.1180 total 0.1729
cascade + cache + semantic cache gen 0.0078 check 0.0021 human 0.0317 rework 0.1985 total 0.2401
router + escalate + cache gen 0.0140 check 0.0021 human 0.1259 rework 0.0377 total 0.1798For the cascade with prompt caching, generation costs 0.0104 dollars and checking 0.0028 per task. Together that is 0.0132 of a total of 0.1729, or 7.6 percent. The other 92 percent is the human column (0.0417) and the rework column (0.1180). The model spend is the small part. What decides cost per successful task is how often the system hands work to a person and how often it lets a wrong answer through, which depends on the checker and the tier success rates, not on the price per token.
The counterexample: cheaper requests, costlier tasks
The two tables contain the case from the opening. Compare the large tier alone with the small tier retried up to three times behind a checker. The request-level figure favors the retrying policy by a wide margin.
| Policy | \$ per request | Requests per task | \$ per successful task (R = 0) | \$ per successful task (R = 3) |
|---|---|---|---|---|
| L only, no check | 0.0275 | 1.00 | 0.0288 | 0.1690 |
| S x3 retries + check | 0.0040 | 2.99 | 0.6132 | 0.8323 |
| Cascade + prompt cache | 0.0048 | 2.73 | 0.0571 | 0.1800 |
The retrying policy has the lowest cost per request of all nine, 0.0040 dollars, which is 6.9 times below the large tier alone. Its cost per successful task is 21 times higher at \(R = 0\) and 4.9 times higher at \(R = 3\). Two things happen. It makes 2.99 requests per task, so a task is triple the request cost. More importantly, the small tier fails often on medium and hard tasks and retrying it rarely fixes them, so 14 percent of tasks end at a person, and a person costs about as much as a thousand small-tier calls. A dashboard that averages cost over requests rewards this configuration because most of its requests are cheap small-tier calls, including the retries that were wasted.
The cascade is a subtler case. Its cost per request is 5.7 times lower than the large tier alone, yet at \(R = 3\) its cost per successful task is 6.5 percent higher, 0.1800 against 0.1690. It does deliver more correct answers, 96.1 percent against 95.5 percent, and fewer wrong ones, 3.9 percent against 4.5. The extra cost is the escalation path. The cascade sends 1.0 percent of tasks to a person, 0.0417 dollars per task, and spends 0.0028 on checks, while the rework it avoids is only 0.0159 dollars per task (0.1339 for the large tier alone, 0.1180 for the cascade). The checker’s 10 percent false-accept rate limits that saving: the cascade stops at the first tier whose output the checker accepts, so a share of the small and mid tiers’ wrong answers is delivered. The false-accept table below shows how much of the result depends on it.
Retries and agent loops multiply cost
In a chat product each request stands alone. In an agent, a model repeatedly decides on an action, a tool runs, and the result is appended to a growing context for the next decision. Every step re-sends the entire context as input. If the context starts at \(B\) tokens and grows by \(g\) tokens per step, the input tokens billed over \(n\) steps are \(\sum_{i=0}^{n-1}(B + i g) = nB + g\,n(n-1)/2\). The second term is quadratic in the number of steps: doubling the length of a run more than doubles its input cost. The numbers below use 4,000 starting tokens, 1,200 tokens of growth per step, and 300 output tokens per step at mid-tier prices.
agent run cost (one attempt), per-step context growth 1,200 tokens, base 4,000
steps no cache cache ratio input tokens billed (no cache)
1 0.0110 0.0130 0.85 4,000
5 0.0790 0.0416 1.90 32,000
10 0.2180 0.0828 2.63 94,000
20 0.6760 0.1832 3.69 308,000
40 2.3120 0.4560 5.07 1,096,000A forty-step run bills more than a million input tokens even though the task started with four thousand. This is the cost that makes agents expensive, and it is also why caching matters most for them: the previous step’s context is an exact prefix of the next step’s.
The cache column prices that case, with the previous context read from cache and only the new tokens written. At one step caching costs more than not caching (ratio 0.85), because the write premium has no read to pay it back. By twenty steps the cached run costs 3.7 times less. The write premium of 1.25 times is repaid after a single later read, which is the reason caching suits loops and not one-shot calls.
A second multiplier comes from failures. If each step succeeds with probability \(s\), a run of \(n\) steps succeeds with probability \(s^{n}\), and success decays fast with length. The next table lets a run restart from scratch up to three times, charges each attempt as a full run (an upper bound, since failed runs often stop early), and computes the expected cost of one successful run.
whole-run restarts: per-step success s, run succeeds with s**n, up to k attempts, cost per successful run
steps s run succ attempts P(any ok) $ per success
10 0.99 0.904 1.10 0.999 0.0916
10 0.95 0.599 1.56 0.935 0.1384
20 0.95 0.358 2.05 0.736 0.5111
20 0.90 0.122 2.65 0.322 1.5072A ten-step agent with 99 percent per-step reliability costs 0.0916 dollars per success and completes 99.9 percent of the time with up to three attempts. At 95 percent per step, the same ten steps succeed in only 59.9 percent of single runs and cost 0.1384 dollars per success. A twenty-step agent at 90 percent per step succeeds in 12.2 percent of runs, and even with three attempts only 32.2 percent of tasks finish, at 1.51 dollars per finished task. Reliability per step moves cost per successful task much faster than price per token does, and the effect grows with run length. A hard cap on steps and on spend per task is therefore a cost control as well as a safety control. The previous article on service objectives treated task success as a first-class objective, and this is the economic reason.
Caching
Prompt caching
Prompt caching reuses the computation for a prompt prefix that an earlier request already processed. The provider stores the processed prefix, matches the new prompt against it from the start, and charges the reduced read price for the matched part. The documented mechanism has several conditions that decide whether the savings appear.
- Prefix match. Reuse works on an exact prefix. The first differing token ends the match, so content that changes per request, such as a timestamp or a user name, must come after the stable part. Putting a timestamp at the top of a system prompt makes every request a miss.
- Breakpoints and lookback. Caching is requested by marking a content block with a
cache_controlfield, or by a single top-level field that moves the breakpoint forward as a conversation grows. The system checks only a limited number of earlier blocks, 20 per breakpoint in the documentation, when looking for a match. - Minimum length. A prefix shorter than the model’s minimum is processed without caching and without an error. The minimums in the documentation range from 512 to 4,096 tokens by model, which is why the 3,000-token prefix in the simulation caches on the mid and large tiers and not on the small one.
- Lifetime. The default lifetime is five minutes, refreshed free on each use. A one-hour option costs twice the input price to write. Traffic that arrives less often than the lifetime pays the write premium repeatedly and rarely reads.
- Invalidation. Changing tool definitions, toggling certain features, or editing earlier content invalidates the cache from that point on.
Whether caching saves money depends on the hit rate \(h\), the fraction of requests that find their prefix already cached. With the five-minute multipliers, the expected prefix cost relative to no caching is \(0.1h + 1.25(1-h)\). It falls below 1 when \(h > 0.25/1.15 \approx 0.217\).
prompt-cache break-even on hit rate h (5-minute write 1.25x, read 0.1x of input price)
h=0.10 expected prefix cost = 1.135 x uncached
h=0.20 expected prefix cost = 1.020 x uncached
h=0.22 expected prefix cost = 0.997 x uncached
h=0.30 expected prefix cost = 0.905 x uncached
h=0.50 expected prefix cost = 0.675 x uncached
h=0.90 expected prefix cost = 0.215 x uncached
h=0.95 expected prefix cost = 0.158 x uncached
break-even h = 0.2174At a 10 percent hit rate caching costs 13.5 percent more than not caching on the prefix. At 95 percent it costs 16 percent of the uncached price. The practical consequence is that cache hit rate belongs on the dashboard next to cost, as cache_read_input_tokens and cache_creation_input_tokens from the provider’s usage fields over total input tokens. A deployment that reads almost nothing and writes constantly is paying a premium for a feature that is not working. In the cascade simulation, enabling prompt caching reduced cost per successful task from 0.0602 to 0.0571 at \(R = 0\), a 5 percent change. Prompt caching is large only when the repeated prefix is a big share of the input and of the total cost. For short outputs on long prompts and for agent loops it is the largest lever. For tasks that are dominated by human review it barely registers.
Semantic caching
A semantic cache stores complete answers and returns a stored answer when a new request is similar enough to an earlier one, usually judged by embedding similarity. It saves the whole generation cost, not just the input discount, and it can cut latency to a lookup. It is also the caching approach with the most ways to be wrong. The glossary entry on caching in machine learning makes the point that the useful question is whether a cached value is still valid for this request, and an approximate match can fail that test while looking like a hit.
- Similar is not the same. Two questions that differ in one entity, one date, or one negation embed close together and need different answers. A threshold loose enough to give a high hit rate will serve some of them wrong answers.
- Identity. The key must include everything the answer depends on: the model version, the prompt version, the retrieved documents, and the tenant. This is the isolation concern from the previous article. A cache shared across tenants can return one customer’s answer to another, so the cache must be scoped to the same boundaries as the data.
- Staleness. A stored answer reflects the knowledge, policy, and data of the day it was made. A time-to-live bounds the age but cannot detect that a source changed an hour after the answer was stored.
- Permissions. An answer produced for a user who could see a document must not be served to one who cannot. The permission check has to run at read time against current permissions.
In the simulation, adding a semantic cache that answers 25 percent of tasks with an 85 percent correct stored answer lowers requests per task from 2.73 to 2.04 and generation spend per task from 0.0104 to 0.0078 dollars. At \(R = 0\) cost per successful task falls from 0.0571 to 0.0445, which looks like a clear win. At \(R = 3\) it rises from 0.1800 to 0.2571, because the cache answers 25 percent of tasks without a checker, and 15 percent of those answers are wrong, so wrong deliveries rise from 3.9 to 6.6 percent. The same mechanism produces both outcomes. A semantic cache pays off only when wrong hits are rarer or cheaper than the generations they replace, and the break-even depends on the hit threshold, so it should be measured on audited samples of served hits.
Routing
Routing sends each task to a tier chosen by cost and expected quality. Two designs appear in practice. A cascade tries the cheapest tier first and escalates when a check rejects the output. A router predicts in advance which tier a task needs and sends it there, optionally with escalation as a safety net. The model gateway from the earlier article is the natural place for either, since it already sees every request and holds the tier definitions.
A cascade pays for its failures. Every task the small tier cannot do costs a small-tier call and a check before the mid tier is tried. If the small tier fails on 40 percent of tasks, those tasks cost more than if they had gone to the large tier directly. A router avoids this when it predicts well, and pays a classification cost on every task and a misrouting penalty when it is wrong. In the simulation the router reaches a success rate of 0.987 against the cascade’s 0.961, but it spends more on people, 3.1 percent of tasks against 1.0 percent. A task routed directly to the large tier has one attempt before a person is called, while a cascade gives every task up to three. The router’s success rate is higher because fewer wrong answers escape, 1.3 percent against 3.9: hard tasks skip the small tiers, whose mistakes the checker would sometimes pass. At \(R = 3\), the two have nearly equal cost per successful task, 0.1821 and 0.1800.
The checker’s accuracy decides how well either design works. The next table varies the false-accept rate from 2 to 30 percent at \(R = 3\) and reports cost per successful task.
checker false-accept rate sweep, R = $3.00 ($ per successful task)
FAR L only, no check cascade + cache router + cache
0.02 0.1690 0.0917 0.1608
0.05 0.1690 0.1227 0.1693
0.10 0.1690 0.1800 0.1821
0.20 0.1690 0.2957 0.2114
0.30 0.1690 0.4153 0.2381With a checker that accepts only 2 percent of wrong outputs the cascade costs 0.0917 dollars per success, 46 percent below the large tier alone at 0.1690. At 10 percent it is 6.5 percent above, and at 30 percent it is nearly 2.5 times higher. The cascade is not a cost optimization on its own. It is a cost optimization conditional on a verifier that can tell good from bad cheaply. Where no such verifier exists, for example in open-ended writing without ground truth, escalating on a judge’s opinion mostly moves money around. The router degrades more slowly as the checker worsens because it relies on the checker less often, which suggests that routers are more robust where verification is weak and cascades where verification is strong.
The quality-cost frontier
The tables so far show that no policy is best everywhere. A Pareto frontier formalizes this: a policy is on the frontier if no other policy has both lower cost per task and a higher success rate. Everything off the frontier is dominated and can be dropped. Everything on it is a legitimate choice, and choosing among them is a business decision about how much a point of success rate is worth, not a technical one.
The quantity that decides is \(R\), the cost of a wrong delivered answer, together with the human cost \(H\). The next table sweeps \(R\) and reports cost per successful task for four policies.
cost of a delivered wrong answer R vs $ per successful task
R S only M only L only cascade+cache
0.00 0.0076 0.0156 0.0288 0.0571
0.05 0.0295 0.0220 0.0312 0.0592
0.10 0.0514 0.0285 0.0335 0.0612
0.25 0.1171 0.0479 0.0405 0.0674
0.50 0.2265 0.0802 0.0522 0.0776
1.00 0.4455 0.1449 0.0755 0.0981
3.00 1.3212 0.4037 0.1690 0.1800The smallest tier is cheapest only when wrong answers cost almost nothing. The simulation also computes the exact break-even points, using the fact that for a fixed policy the cost per success is linear in \(R\).
break-even R (S vs M, M vs L, L vs cascade+cache): 0.026 0.161 4.904The small tier alone beats the mid tier alone only below \(R \approx 0.03\) dollars. The mid tier beats the large tier alone only below \(R \approx 0.16\). The cascade with caching beats the large tier alone only above \(R \approx 4.90\). These thresholds are the useful output of the exercise. They convert the debate about which model is cheaper into a question the product owner can answer: what does a wrong answer cost here, in the unit of dollars. A feature where wrong answers are cheap and visible, such as a draft suggestion that the user reads before using, sits at the low end of the table and favors small models. A feature where wrong answers trigger irreversible actions sits at the high end and favors strong models, verification, or removing the model from the decision.
Policies can also be compared at fixed quality. If the product requires a success rate of 0.98, then in this simulation only the large tier with a checker and the router qualify, at 0.3176 and 0.1798 dollars per task at \(R = 3\). The router is 43 percent cheaper at equal or better quality. Fixing the quality target first and then minimizing cost avoids the failure of the opening example, where cost was minimized and quality was left to chance.
Limits of these numbers
Several limits apply to every figure above.
- Prices change. Providers have changed list prices, introduced new tiers, and adjusted cache multipliers between model generations; the pricing page itself shows three different cache-read multipliers across current models. Model unit prices as data with a date and version, and recompute rather than reuse a stored dollar figure.
- The workload is synthetic. The difficulty mix, the success probabilities, the checker error rates, and the reviewer cost are inventions that make the arithmetic concrete. Real success probabilities are measured on a representative labeled set, and they vary with prompt, tool design, and the version of the model.
- Failures correlate. The simulation reduces retry success to 0.6 of the first attempt’s probability, a modeling choice. Independent retries would make the retrying policy look better than it is. Correlation is real but its size depends on the task.
- Measurement biases the ratio. The denominator comes from a checker whose errors skew the observed rate, and \(R\) is an estimate that is rarely measured. Small changes in either can reorder policies near the break-even points, which is why the results are better read as ranges of \(R\) than as a single winner.
- Not modeled. Latency, which affects both user experience and the cost of waiting. Rate limits and queueing. Output-length changes when a prompt changes. Volume discounts and committed-use contracts. Fixed platform costs such as engineering time and observability, which belong in a full cost-of-ownership calculation but not in a per-task marginal one.
Putting the metric into operation
To compute cost per successful task in production, the platform needs a few things recorded consistently. The unit of record is the task, not the request. A task identifier generated at the entry point and propagated to every model call, retry, tool call, and judge call lets the gateway sum cost across all of them. A record per task might look like the following. It is a schema example, not output from a run.
{
"task_id": "t-7f3a",
"tenant": "tenant-a",
"application": "ticket-assistant",
"release": {"model_alias": "mid-v12", "prompt": "p-41", "index": "i-2024-11-20"},
"requests": 3,
"cost_usd": {"generation": 0.0102, "check": 0.0028, "human": 0.0, "tools": 0.0},
"tokens": {"input": 7010, "cache_read": 5850, "cache_write": 0, "output": 912},
"outcome": "delivered",
"check": {"verdict": "accepted", "method": "schema+judge"},
"audited_correct": null
}The audited_correct field is empty for most tasks and filled for a sampled fraction by a human or a trusted automated check. It is what turns the checker’s verdict into a calibrated estimate, and it is the number to use for the denominator. The release block ties each cost to the model alias, prompt, and index versions in the artifact versioning scheme, so a change in cost per success can be traced to a change that caused it. The telemetry itself must follow the rules from the article on observing AI systems without exposing sensitive data: cost and outcome fields carry no prompt content.
With those records, four operating practices make the metric usable.
- Report cost per successful task alongside cost per request. When the two move in opposite directions, the request-level figure is misleading. That divergence is a signal that retries, escalations, or wrong deliveries are changing.
- Bound the multipliers. Set a maximum number of attempts, a maximum number of agent steps, and a maximum spend per task at the gateway, so a loop or a retry storm hits a ceiling instead of a monthly budget.
- Gate releases on the cost-quality pair. A candidate that lowers cost per request but raises cost per successful task on the evaluation set should not pass, and the evaluation gate can compute both from the same run.
- Watch cache hit rate and the audited wrong-hit rate separately. The first is a cost metric and the second is a quality metric, and a cache tuned only for the first can degrade the second without any alert.
These controls assume things work as intended. Sometimes they do not: a release raises cost and wrong answers together, a provider degrades, or a model takes an action it should not have. Cost spikes are among the earliest signals of such events, since a retry storm or a runaway agent shows up in spend before it shows up in complaints. The next article, AI Platform Incident Response and Recovery, covers what to do then: how to contain bad outputs and unsafe actions, how to roll back model and prompt versions, and how to verify that recovery worked.
References
- Anthropic documentation, Pricing
- Anthropic documentation, Prompt caching
- Anthropic documentation, Batch processing
- FinOps Foundation, FinOps Framework
- Chen, Zaharia, and Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.