Optimizing AI Applications for Cost and Latency
Consider a hypothetical team that runs the returns assistant from the earlier articles. Traffic has grown, the monthly bill has grown with it, and p95 latency sits just under its budget. Three proposals arrive in one week: send easy requests to a smaller model, cache repeated work, and cut the retrieved context from eight passages to five. Each is plausible, and each is a bet. It moves cost, latency, and quality at the same time, by different amounts for different kinds of requests. A team that adopts all three because the average cost per request fell will find that some of the savings were real, some were canceled by a rise in failures, and one made the slowest requests slower.
Deciding well takes four things. The first is a unit of cost that charges failures to the system, so that a cheap call that fails does not look like a saving. The second is an understanding of what routing, caching, and context reduction each reuse or remove, and under which conditions that is safe. The third is a view of latency as a tail made of components that compound. The fourth is an evaluation plan that is itself affordable and controlled, because choosing among a dozen configurations is a measurement problem with its own cost. The measurements and stage discipline of online evaluation and controlled rollout, the acceptance criteria written during task framing, and the limits and timeouts of model API integration carry over unchanged and are used as given.
The programs are Python 3.12 using only the standard library, and everything they run is synthetic. Prices are abstract cost units written u, times are milliseconds, and every success, latency, and confidence score is drawn from a configured distribution, so the output shows how a method behaves and says nothing about any provider, model, or product. Real values have to be measured on the application’s own traffic.
Three quantities and one decision
Quality, latency, and cost do not share a unit, and they should not be forced into one early. The framing article already gave each of them a form that can be checked. Quality is a floor per request class, because an average lets a large common class hide a collapse in a rare critical one. Latency is a percentile of the end-to-end time with a deadline and a fallback. Cost came as a cost matrix that prices each kind of error. Optimization then has the shape of a constrained problem: minimize cost subject to every class floor and the latency budget.
Two consequences follow. A configuration \(A\) dominates a configuration \(B\) when \(A\) is no worse on every class floor, on the latency percentile, and on cost, and strictly better on at least one. Dominated configurations can be discarded without any judgment about priorities, and what remains is the frontier, the set of real trade-offs. And a single weighted score such as “quality minus cost” hides which class or which percentile pays for a saving, so it comes last, after the constraints have removed the configurations that violate them.
The synthetic world used throughout
The returns assistant is simulated by world.py. A request belongs to one of five classes, with a share of traffic and a difficulty \(d \in [0,1]\) drawn from a class-specific Beta distribution; the conflict class is the hardest. A model has a capability \(c_m\), and its probability of success on a request is a logistic function of the margin, \(p = 1/(1+e^{-10(c_m – d)})\). The evidence the answer needs sits at some rank in the retrieved list, or is never retrieved in 6% of requests. A call is priced by tokens, with the large model ten times the small model’s price per token, and its latency is a fixed part, a part proportional to input tokens, a part proportional to output tokens, multiplicative noise, and the 3% slow path that the framing article used for its latency model.
import math, random
from dataclasses import dataclass
# class: (traffic share, Beta(a, b) of request difficulty d in [0, 1])
CLASSES = {"standard": (.55, 2, 5), "late": (.15, 2, 4), "final_sale": (.10, 2.5, 3.5),
"missing_info": (.12, 3, 3), "conflict": (.08, 4, 2.5)}
# hypothetical prices in cost units (u) per 1,000 tokens; times in ms; cap = difficulty a model handles
MODELS = {"small": dict(cap=.55, in_u=.1, out_u=.4, base=120, per_in=.04, per_out=5),
"large": dict(cap=.85, in_u=1.0, out_u=4.0, base=300, per_in=.10, per_out=14)}
SLOPE, PREFIX_TOK, PASSAGE_TOK, OUT_TOK = 10, 900, 220, 80
SLOW_P, SLOW_MS = .03, 1200 # slow path: 3% of calls add an exponential delay, mean 1.2 s
@dataclass(frozen=True)
class Req:
id: int; cls: str; d: float; u: float; gold_rank: int; q_tok: int
def make_requests(n, seed, first_id=0):
rng = random.Random(seed)
names = list(CLASSES); shares = [CLASSES[0] for c in names]
out = []
for i in range(n):
c = rng.choices(names, shares)[0]
_, a, b = CLASSES
gold = 99 if rng.random() < .06 else 1 + min(15, int(rng.expovariate(1.0))) # 99: evidence never retrieved
out.append(Req(first_id + i, c, rng.betavariate(a, b), rng.random(), gold, rng.randint(40, 160)))
return out
def p_success(model, r, k):
p = 1 / (1 + math.exp(-SLOPE * (MODELS[model]["cap"] - r.d)))
if r.gold_rank > k:
return .15 * p # without the evidence, only occasional success
return p * (1 - .008 * (k - 1)) # each extra passage adds distraction (assumed)
def call(model, r, k, attempt=0):
"""One model call: returns (success, cost_u, latency_ms, confidence). Deterministic per (request, model, attempt)."""
m, rng = MODELS[model], random.Random(f"{r.id}:{model}:{attempt}")
ok = r.u < p_success(model, r, k) # r.u is the case's own luck, shared by every configuration
tin = PREFIX_TOK + r.q_tok + k * PASSAGE_TOK
cost = tin / 1000 * m["in_u"] + OUT_TOK / 1000 * m["out_u"]
lat = (m["base"] + m["per_in"] * tin + m["per_out"] * OUT_TOK) * rng.lognormvariate(0, .25)
if rng.random() < SLOW_P:
lat += rng.expovariate(1 / SLOW_MS)
conf = min(1, max(0, rng.gauss(.72 if ok else .45, .14))) # hypothetical self-check score
return ok, cost, lat, conf
def pct(xs, q):
xs = sorted(xs); return xs[min(len(xs) - 1, int(q * len(xs)))]Each case carries its own random draw u, and a call succeeds when u is below the model’s success probability. Every configuration therefore sees the same luck on the same case, which is what makes the comparisons later in the article paired. It also nests the outcomes: whenever the small model succeeds, the large model succeeds too, which is optimistic, because real models fail on different cases. Latency noise is seeded by request, model, and attempt, so reruns reproduce. The context penalty (0.8% lower success per extra passage) is an assumption of the toy, set to make the effect of context size visible, and a real curve has to be measured for each class and model. Queueing is not simulated: each call sees an unloaded service.
Cost per successful task
A bill divided by requests is the number a dashboard shows first, and it misleads for a reason that can be written down. Let \(c\) be the average spend per request, \(s\) the share of requests that succeed, and \(F\) the expected cost of one failed task: rework by a person, a second contact, an escalation, or a harm priced as in the cost matrix. Three quantities follow:
- Spend per request, \(c\), what the provider and the infrastructure charge.
- Spend per successful task, \(c/s\), which charges the money wasted on failures to the successes.
- Total cost per task, \(c + (1-s)F\), which adds what the failures cost downstream.
The second quantity rises as quality falls, and it is the one that vendors’ price comparisons and many internal reports stop at. The third is the one a decision needs, because a failed task still has to be handled. Comparing a cheap model \(S\) with an expensive model \(L\) on the third quantity gives a break-even condition. The cheap model has the lower total cost when \(c_S + (1-s_S)F < c_L + (1-s_L)F\), which rearranges to
\[ F < F^{*} = \frac{c_L - c_S}{s_L - s_S}. \]
In words: the cheap model wins only if a failure costs less than the extra spend divided by the extra success rate the expensive model buys. The price gap and the quality gap are both averages over the traffic mix, so the result holds for the mix that was measured. The program below applies the three definitions to the two models on 20,000 simulated requests with five passages per prompt and \(F = 40\) u, an assumption standing for a few minutes of staff time.
from world import *
from collections import defaultdict
F = 40.0 # hypothetical expected cost of one failed task (rework, escalation, or harm), in u
reqs = make_requests(20000, seed=1)
print(f"{'model':6s}{'success':>9s}{'spend/req':>10s}{'spend/success':>14s}{'total/task':>11s}{'p95 ms':>8s}")
stats = {}
for m in ("small", "large"):
res = [call(m, r, 5) for r in reqs]
s = sum(x[0] for x in res) / len(res); c = sum(x[1] for x in res) / len(res)
stats[m] = (s, c)
print(f"{m:6s}{s:9.3f}{c:10.3f}{c/s:14.3f}{c+(1-s)*F:11.3f}{pct([x[2] for x in res], .95):8.0f}")
(s_s, c_s), (s_l, c_l) = stats["small"], stats["large"]
print(f"\nbreak-even failure cost F* = (c_large - c_small) / (s_large - s_small) = {(c_l-c_s)/(s_l-s_s):.1f} u")
by = defaultdict(lambda: [0, 0, 0])
for r in reqs:
for i, m in enumerate(("small", "large")):
by[r.cls][i] += call(m, r, 5)[0]
by[r.cls][2] += 1
print("\nsuccess by class small large gap")
for c, (a, b, n) in by.items():
print(f"{c:14s} {a/n:8.3f}{b/n:7.3f}{(b-a)/n:7.3f}")model success spend/req spend/success total/task p95 ms
small 0.704 0.242 0.344 12.068 978
large 0.880 2.420 2.748 7.204 2574
break-even failure cost F* = (c_large - c_small) / (s_large - s_small) = 12.4 u
success by class small large gap
standard 0.789 0.905 0.116
conflict 0.352 0.745 0.393
final_sale 0.643 0.870 0.227
missing_info 0.538 0.845 0.307
late 0.744 0.894 0.151The small model spends 0.242 u per request and the large one 2.420 u, so the small model is ten times cheaper per request. Per successful task the ratio is 0.344 u against 2.748 u, still a factor of eight in favor of the small model, because it fails more often (success 0.704 against 0.880). Once the failures are charged, the totals are 12.068 u and 7.204 u, and the order flips. The break-even failure cost is 12.4 u, so the small model is the cheaper choice only where an error costs less than about a third of what was assumed. The class table shows where the gap comes from. The two models are 12 points apart on standard requests and 39 points apart on conflicts. A single choice of model for all traffic pays the large model’s price on the requests where the small one would have been fine, or the small model’s failures on the requests where it was not. That asymmetry is what routing exploits.
What goes into the numerator and the denominator
The numerator is every unit spent to handle the task, including the spend that did not help. Retries after a timeout, repair calls after a schema failure, guard and judge calls, embedding and retrieval calls, a reranker’s per-candidate cost (see hybrid retrieval and reranking), tool calls, and the steps of an agent loop all belong to it, as does the cost of a person who reviews or approves (see human approval and escalation). An agent run that ends in a budget stop (see bounded execution) consumed its spend and produced no success, so its cost lands on the other runs’ successes. Provisioned capacity belongs there too when the model is self-hosted, and the glossary entry on inference cost explains why the denominator, whether requests, tokens, or completed tasks, has to be stated before two costs are compared.
The denominator is the number of tasks that succeeded, and “succeeded” is the matured outcome defined in the online-evaluation article: a return completed correctly and no contact about the same order within seven days, not the model’s own claim and not a thumbs-up. Until that outcome has matured, cost per successful task is an estimate computed on a cohort that is a week old. The same article’s guardrails on spend and latency watch the lower rungs of the measurement ladder, and the matured number is what confirms that a saving was a saving.
Prices themselves are inputs that change. A provider’s price list has separate rates for input, output, and sometimes cached input, and it changes over time, so a cost model keeps the price table as data and recomputes \(F^{*}\) and the chosen configuration when it changes. The prices in this article are hypothetical for that reason.
Routing: spend the large model where it changes the outcome
Routing sends each request to one of several models, or to a model plus a fallback, so that expensive capacity is used on the requests where it changes the outcome. Two designs are common. A router decides before any answer exists, from the request alone: a rule on request class, a classifier of difficulty, or a learned scorer. A cascade decides after a cheap answer exists: the small model answers, a check estimates whether the answer is right, and the request is escalated to the large model when the check says no. Chen, Zaharia, and Zou describe the cascade among three cost-reduction strategies in FrugalGPT and report that a learned cascade can match the best single model on their benchmarks with up to 98% lower cost. Ong and colleagues train routers on human preference data to choose between a strong and a weak model and report cost reductions of more than a factor of two in some settings without a measured loss of response quality. Both results depend on the task mix, the model pair, and the benchmark, which is the reason the decision below is made on the application’s own measurements.
The cascade has a clean accounting. Let \(e\) be the share of requests that are escalated. The expected spend is \(c_S + e\,c_L\), since every request pays for the small call and escalated ones also pay for the large one. Quality is the share of requests that the check accepts and the small model got right, plus the escalated requests that the large model gets right. The check is a classifier of correctness, and it makes two kinds of mistakes. A false accept lets a wrong small answer through, and it costs quality. A false escalation re-answers a correct small answer, and it costs spend and time but not quality. The threshold \(\tau\) moves the balance. Latency adds as well, because the two calls are sequential: an escalated request pays for both, so it lands in the slow part of the distribution by construction. That is the chain effect from the framing article, now chosen by the application’s own design.
A router has the opposite profile. It adds one cheap prediction to every request and never runs two models, so its latency is that of the model it chose, and its mistakes are of two kinds as well: a hard request sent to the small model fails, and an easy one sent to the large model wastes spend. A router’s input is the request, so it cannot see how the model actually behaved on it, and a check can. The program compares both on a development sample of 6,000 requests and then confirms the chosen policy on a fresh sample. The router’s difficulty estimate is the true difficulty plus Gaussian noise of 0.15; the check score is the self-assessment drawn in world.py, which is higher on average for correct answers (0.72) than for wrong ones (0.45). A router cost of 0.01 u and 25 ms is assumed.
import random
from world import *
F, K, ROUTER_U, ROUTER_MS = 40.0, 5, .01, 25 # failure cost; passages; router cost and latency (assumed)
def run(policy, reqs):
ok = spend = esc = 0; lats = []
for r in reqs:
if policy[0] == "small": o, c, l, _ = call("small", r, K)
elif policy[0] == "large": o, c, l, _ = call("large", r, K)
elif policy[0] == "router": # predict difficulty with noise, send hard requests to large
dhat = r.d + random.Random(f"router:{r.id}").gauss(0, .15)
o, c, l, _ = call("small" if dhat < policy[1] else "large", r, K)
c += ROUTER_U; l += ROUTER_MS
else: # cascade: accept the small answer if its check score >= tau
o, c, l, conf = call("small", r, K)
if conf < policy[1]:
o2, c2, l2, _ = call("large", r, K); o, c, l = o2, c + c2, l + l2; esc += 1
ok += o; spend += c; lats.append(l)
n = len(reqs)
run.esc = esc / n # share of requests that reached the large model in a cascade
return ok / n, spend / n, spend / n + (1 - ok / n) * F, pct(lats, .5), pct(lats, .95)
dev, test = make_requests(6000, 11), make_requests(6000, 12, first_id=100000)
pols = [("small",), ("large",)] + [("router", t) for t in (.4, .5, .6, .7)] + [("cascade", t) for t in (.4, .5, .55, .6, .7)]
print(f"{'policy':16s}{'success':>8s}{'spend':>7s}{'total':>7s}{'p50':>6s}{'p95':>6s}{'escal.':>8s}")
for p in pols:
s, c, tot, p50, p95 = run(p, dev)
print(f"{p[0] + (' %.2f' % p[1] if len(p) > 1 else ''):16s}{s:8.3f}{c:7.3f}{tot:7.3f}{p50:6.0f}{p95:6.0f}{(f'{run.esc:8.2f}' if p[0]=='cascade' else ' -')}")
best = min(pols[2:], key=lambda p: run(p, dev)[2])
print("\nlowest total on dev:", best)
for name, rs in (("dev", dev), ("test", test)):
s, c, tot, p50, p95 = run(best, rs); sl, cl, tl, _, pl = run(("large",), rs)
print(f"{name:5s} chosen: success {s:.3f} spend {c:.3f} total {tot:.3f} p95 {p95:.0f} | large: success {sl:.3f} spend {cl:.3f} total {tl:.3f} p95 {pl:.0f}")policy success spend total p50 p95 escal.
small 0.697 0.242 12.349 610 978 -
large 0.873 2.420 7.513 1649 2564 -
router 0.40 0.830 1.167 7.954 839 2316 -
router 0.50 0.805 0.874 8.680 724 2208 -
router 0.60 0.774 0.634 9.668 676 2035 -
router 0.70 0.740 0.458 10.871 654 1777 -
cascade 0.40 0.761 0.529 10.069 636 2431 0.12
cascade 0.50 0.811 0.808 8.355 676 2766 0.23
cascade 0.55 0.832 0.996 7.709 710 2905 0.31
cascade 0.60 0.849 1.206 7.239 787 2989 0.40
cascade 0.70 0.868 1.712 6.999 1918 3135 0.61
lowest total on dev: ('cascade', 0.7)
dev chosen: success 0.868 spend 1.712 total 6.999 p95 3135 | large: success 0.873 spend 2.420 total 7.513 p95 2564
test chosen: success 0.864 spend 1.655 total 7.095 p95 3083 | large: success 0.872 spend 2.420 total 7.560 p95 2551Read the table by columns. Always-small gives 0.697 success at 0.242 u, and always-large gives 0.873 at 2.420 u; the policies in between are different ways of buying back the gap. At similar spend the cascade is ahead of the router: the router at threshold 0.40 spends 1.167 u for 0.830 success, while the cascade at 0.55 spends 0.996 u for 0.832. The check sees the answer, and the router sees only a noisy view of the request. Minimizing total cost with \(F=40\) u selects the cascade at \(\tau=0.70\) on the development sample. On the fresh sample it spends 1.655 u against 2.420 u for the large model, a 32% saving, at a success rate of 0.864 against 0.872.
The latency columns carry the other half of the result. At \(\tau=0.70\), 61% of requests are escalated, so the median request now pays for two calls: the p50 is 1,918 ms against 1,649 ms for always-large, and the p95 is 3,135 ms against 2,564 ms. The policy that wins on cost loses on both latency percentiles. Whether it is acceptable is decided by the budget in the acceptance criteria and not by the cost column. If the budget were a p95 of 2.8 s, the cascade at \(\tau=0.50\) (p95 2,766 ms, spend 0.808 u) would fit, and its quality would be 0.811, which has to clear the class floors on its own. The cost-minimizing policy is only the answer to the question asked with the latency constraint left out.
What goes wrong with routing
- The check is not calibrated. A self-reported confidence or a score derived from token probabilities is not a probability of correctness by default, and it shifts when the model, the prompt, or the traffic changes (see probability calibration). A threshold chosen on last quarter’s scores can escalate half as often or twice as often after a model update, so the escalation rate is monitored and the threshold is re-fitted with every model change.
- Critical classes need their own rule. Class floors are not met on average. A false accept costs most where the cost matrix says errors are expensive, so classes with a high error cost can be pinned to the large model by a rule, and the cascade applies to the rest. The per-class table in the cost section shows that the cheap model’s deficit is concentrated in exactly the harder classes.
- The router and the check are components with their own error rates. They are evaluated as classifiers (recall of “needs the large model” per class) on the same labeled sample as the end-to-end quality, with the same paired and stratified procedure from the reproducible evaluation article.
- Capacity moves. A cascade that escalates 61% of traffic sends 61% of requests to the large model’s rate limit and queue, plus the small model’s load on top. The rate-limit arithmetic in the model API article is repeated for each model in the route.
- Thresholds can be learned online. Allocating requests among options with exploration, when feedback arrives continuously, is the subject of bandits and exploration; for a delayed task outcome the same caution about immature labels as in the online-evaluation article applies.
Caching: reuse the work, not the wrong answer
A cache returns stored work instead of repeating it. Three caches differ in what they store, what makes a stored item valid, and what a wrong hit costs, and a team that treats them as one feature picks up the risk of the most dangerous one.
| Cache | What is reused | Valid when | Savings | Failure of a wrong hit |
|---|---|---|---|---|
| Prefix (context) cache | The provider’s processed form of an identical prompt prefix | The prefix is token-for-token identical and within its lifetime | Input charges and time to first token | None for the answer: the model still runs on every request |
| Result cache | The final answer to a repeated request | The answer depends on nothing that changed: query, policy and corpus version, prompt release, caller’s permissions | The whole call | A stale or unauthorized answer is served with no model call to catch it |
| Semantic cache | The answer to a similar earlier request | Similarity implies the same correct answer | The whole call, for paraphrases | A silently wrong answer to a question that only looks the same |
Prefix caching
Many requests begin with the same long text: instructions, examples, a policy manual. A service that has already processed that prefix can keep the result and apply it to a later request that begins identically, so the shared part costs less and the first token comes sooner. The mechanism inside the model is the key-value cache of the prefix, covered in the article on LLM inference optimization, and the glossary entry on context caching summarizes the application-level view. One provider’s documentation (Anthropic’s prompt caching guide) describes the shape that is common: prefixes up to marked breakpoints are cached, cache reads are billed at a fraction of the base input price, cache writes at a premium, the default lifetime is five minutes and is refreshed each time the entry is used, and the response reports how many input tokens were read from and written to the cache. The concrete numbers differ by provider, model, and over time.
A provider-neutral cost model needs four parameters. The prefix has \(P\) tokens and the per-request part \(V\) tokens. A cache write costs \(w\) times the base input price and a cache read costs \(r\) times it, with \(w>1>r\). If a share \(h\) of requests find the prefix cached, the input cost relative to no caching is
\[ \frac{P\,\bigl(h\,r + (1-h)\,w\bigr) + V}{P + V}. \]
In words: the prefix is charged at the read price on a hit and at the write price on a miss, and the per-request part is charged at full price either way. The prefix pays off when \(h\,r + (1-h)\,w < 1\), that is when the hit rate exceeds \(h^{*} = (w-1)/(w-r)\). Because an entry lives for a time-to-live \(T\) that each use refreshes, and requests that share the prefix arrive at rate \(\lambda\), a Poisson stream finds the prefix cached with probability \(h = 1 - e^{-\lambda T}\), the chance that the previous request was less than \(T\) ago. The program checks that formula by simulation and prices a 3,000-token prefix with 500 tokens per request, using hypothetical multipliers \(w = 1.25\) and \(r = 0.10\) and \(T = 5\) minutes.
import math, random
P, V = 3000, 500 # shared prefix tokens, per-request tokens
W, R, TTL = 1.25, .10, 5.0 # hypothetical: write premium, read discount (x base input price), TTL in minutes, refreshed on use
print(f"break-even hit rate h* = (W-1)/(W-R) = {(W-1)/(W-R):.3f}")
print(f"{'req/min':>8s}{'hit rate (sim)':>16s}{'1-exp(-lam*TTL)':>17s}{'input cost vs no cache':>24s}{'with a timestamp at the top':>29s}")
rng = random.Random(3)
for lam in (.05, .1, .2, .5, 2, 10):
t, last, hits, n = 0.0, None, 0, 20000
for _ in range(n):
t += rng.expovariate(lam)
hits += last is not None and t - last <= TTL # prefix still cached: any request within TTL of the previous one
last = t
h = hits / n
cost = (P * (h * R + (1 - h) * W) + V) / (P + V)
cost_ts = (P * W + V) / (P + V) # prefix never matches, every request pays the write premium
print(f"{lam:8.2f}{h:16.3f}{1-math.exp(-lam*TTL):17.3f}{cost:24.3f}{cost_ts:29.3f}")break-even hit rate h* = (W-1)/(W-R) = 0.217
req/min hit rate (sim) 1-exp(-lam*TTL) input cost vs no cache with a timestamp at the top
0.05 0.218 0.221 0.999 1.214
0.10 0.394 0.393 0.826 1.214
0.20 0.623 0.632 0.600 1.214
0.50 0.924 0.918 0.304 1.214
2.00 1.000 1.000 0.229 1.214
10.00 1.000 1.000 0.229 1.214The break-even hit rate is 0.217. At 0.05 requests per minute the prefix expires before it is reused about 78% of the time, the input cost is 0.999 of the uncached cost, and the cache is a wash. At 0.2 requests per minute the hit rate is 0.623 and the input cost is 0.600. At two requests per minute and above the hit rate is effectively 1 and the cost reaches the floor \((P\,r + V)/(P+V) = 0.229\), the read price of the prefix plus the full price of what changes. The simulated hit rates agree with \(1-e^{-\lambda T}\) to within sampling error. The last column is the failure mode that costs money: if a timestamp or request id sits above the shared text, the prefix never matches, no read ever happens, and each request pays the write premium, a cost of 1.214 times the uncached input cost. This is why prompt layout is a cost decision: stable text first, per-request text last, and a check of the cached-token counts in the usage report, not an assumption that caching is working.
Prefix caching changes input cost and time to first token only. Output tokens are unaffected, and the model’s behavior on the request is the same, so there is no quality risk to evaluate beyond confirming that the prompt was not reordered to enable it. Text that is specific to one customer does not belong in a prefix that is shared across requests, and what may be placed there is a question for the article on privacy and output controls.
Result caching
A result cache stores the final answer. It saves the whole call, including its tail, and it removes the model from the loop, so every protection that the model’s run provides, such as output checks, must have been applied before the answer was stored. Whether a request is cacheable at all is a property of the request class. A question about the generic return policy has an answer that depends on the policy text and nothing else. A question about the customer’s own order depends on the order system and on who is asking, and the privacy article’s rule that caches do not cross principals applies without exception. In the simulation below, 30% of traffic is cacheable.
What identifies a reusable answer is every input the answer depends on: the normalized request, the version of the corpus or policy text, the prompt release and model, and the permission scope. A key that omits one of them returns an answer that was correct for a different state of the world, and the glossary entry on caching in machine learning makes the same point about versioned keys and about the difference between expiry and invalidation. The program simulates a day of 60,000 requests over 4,000 question intents with Zipf-like popularity, where the policy text changes at hour 12 for 10% of the intents.
import random
HOURS, N, INTENTS = 24, 60000, 4000
CACHEABLE, CALL_U, CHANGE_H = .30, 2.4, 12.0 # share of traffic whose answer does not depend on the customer
rng = random.Random(5)
weights = [1 / (i + 1) ** 1.1 for i in range(INTENTS)] # Zipf-like popularity of question intents
changed = set(range(0, INTENTS, 10)) # 10% of intents are answered differently after the policy change
stream = sorted((rng.uniform(0, HOURS), rng.choices(range(INTENTS), weights)[0], rng.random() < CACHEABLE) for _ in range(N))
def simulate(versioned, TTL_H=6.0):
cache, hits, stale, calls = {}, 0, 0, 0
for t, intent, cacheable in stream:
version = 2 if t >= CHANGE_H and intent in changed else 1 # version of the policy text that applies now
if not cacheable:
calls += 1; continue
key = (intent, version) if versioned else intent
entry = cache.get(key)
if entry and t - entry[0] < TTL_H:
hits += 1; stale += entry[1] != version # served an answer built from an older policy
else:
calls += 1; cache[key] = (t, version)
return hits, stale, calls
print(f"requests {N}, cacheable {CACHEABLE:.0%}, TTL 6 h, policy change for 10% of intents at hour {CHANGE_H:.0f}")
for name, v in (("key = intent only", False), ("key = intent + policy version", True)):
h, s, c = simulate(v)
print(f"{name:30s} hits {h:6d} ({h/N:5.1%} of all) stale answers {s:5d} model calls {c:6d} spend {c*CALL_U:9.0f} u")
print(f"{'no cache':30s} hits 0 ( 0.0% of all) stale answers 0 model calls {N:6d} spend {N*CALL_U:9.0f} u")
print("\nintent-only key, longer TTL:")
for ttl in (1, 6, 24):
h, s, c = simulate(False, ttl)
print(f" TTL {ttl:2d} h hits {h:6d} stale answers {s:5d} ({s/h:5.2%} of hits) spend {c*CALL_U:9.0f} u")requests 60000, cacheable 30%, TTL 6 h, policy change for 10% of intents at hour 12
key = intent only hits 14309 (23.8% of all) stale answers 69 model calls 45691 spend 109658 u
key = intent + policy version hits 14286 (23.8% of all) stale answers 0 model calls 45714 spend 109714 u
no cache hits 0 ( 0.0% of all) stale answers 0 model calls 60000 spend 144000 u
intent-only key, longer TTL:
TTL 1 h hits 11674 stale answers 24 (0.21% of hits) spend 115982 u
TTL 6 h hits 14309 stale answers 69 (0.48% of hits) spend 109658 u
TTL 24 h hits 15775 stale answers 1926 (12.21% of hits) spend 106140 uA result cache with a six-hour lifetime answers 23.8% of all requests, which is 79% of the cacheable ones, and cuts spend from 144,000 u to 109,714 u, a 24% saving. The two keys have the same hit rate and the same saving. They differ in one number. Keying by intent alone served 69 answers built from the old policy after it had changed, and keying by intent and policy version served none, at the price of 23 extra model calls. The TTL block shows that expiry is not a substitute for a version in the key: with a 24-hour lifetime the saving grows to 106,140 u and the stale answers grow to 1,926, which is 12.2% of the hits. A TTL bounds how long an error can last. It does not decide whether the error occurs, and it does not satisfy a rule such as “a withdrawn policy must stop being served now”, which needs an invalidation event.
A result cache also affects the measurements around it. An offline evaluation run that hits the cache measures the cache and not the release, so evaluation either bypasses the cache or keys it by release. A latency or cost estimate from a cold cache understates the saving and one from a warm cache that was filled with the test set overstates it, so the estimate comes from replaying a realistic traffic stream. And, as the online-evaluation article notes, a shared cache is a channel of interference between the arms of an experiment unless the release fingerprint is in the key.
Semantic caching
A semantic cache looks up the nearest earlier request in an embedding space and returns its answer when the similarity passes a threshold. It extends reuse from identical requests to paraphrases, and it replaces exact key equality with a similarity score, which is a matter of degree. An exact hit is safe to reuse only when the key covers the answer’s dependencies, including permission scope, versions, and conversation context, and the stored answer remains valid. Semantic matching adds the risk of treating different requests as equivalent. The risk is in the near neighbors with a different answer. “Can I return this within 30 days?” and “Can I return this after 30 days?” are close in embedding space and have opposite answers. The program draws synthetic similarities for two populations of nearest-neighbor lookups, true paraphrases and near neighbors that need a different answer, and assumes that 80% of lookups are paraphrases.
import random
rng = random.Random(9)
# Synthetic cosine similarities between a new query and its nearest cached query.
same = [rng.gauss(.92, .04) for _ in range(20000)] # paraphrases: the cached answer is correct for the new query
diff = [rng.gauss(.84, .05) for _ in range(20000)] # near neighbors needing a different answer ("within 30 days" vs "after 30 days")
SHARE_SAME = .8 # assumed share of near-neighbor lookups that are true paraphrases
print(f"{'threshold':>9s}{'hit among paraphrases':>23s}{'false hit among different':>27s}{'wrong share of hits':>21s}")
for th in (.80, .85, .88, .90, .92, .95):
h = sum(x >= th for x in same) / len(same); f = sum(x >= th for x in diff) / len(diff)
wrong = (1 - SHARE_SAME) * f / (SHARE_SAME * h + (1 - SHARE_SAME) * f)
print(f"{th:9.2f}{h:23.3f}{f:27.3f}{wrong:21.3f}")threshold hit among paraphrases false hit among different wrong share of hits
0.80 0.999 0.786 0.164
0.85 0.960 0.416 0.098
0.88 0.845 0.211 0.059
0.90 0.696 0.115 0.040
0.92 0.500 0.055 0.027
0.95 0.227 0.013 0.015At a threshold of 0.90, 69.6% of paraphrases hit, and 11.5% of the different-answer neighbors hit as well, so 4.0% of the answers served from the cache are wrong. Raising the threshold to 0.95 brings the wrong share to 1.5% and the paraphrase hit rate to 22.7%. The savings and the error both fall as the threshold rises, and the error never reaches zero in these distributions. A wrong hit raises no error and takes no model call, so it can only be found by auditing a labeled sample of cache hits, with the per-class floors of the acceptance criteria applied to them. A semantic cache is therefore adopted like any other component that can change an answer: with a measured false-hit rate per class, a threshold per class, and no use in classes whose error cost is high. The distributions here are invented, and the share of near neighbors with different answers has to be measured on the application’s own logs.
Reducing context without losing the evidence
Input tokens cost money in proportion to their number and add time before the first output token. The context-and-prompt article set the budget \(E = W – R – I – H – Q\) as a ceiling and noted that more evidence is not automatically better. The question for optimization is how far below the ceiling the application can go before quality falls, and the answer comes from a measurement of quality as a function of the amount of context, per class and per model. The levers are the same ones that build the context: fewer or better-ranked passages, as in the funnel of the retrieval article, a history policy that carries a summary or structured state instead of the transcript, fewer examples, a shorter instruction block, and a compressed passage text. Jiang and colleagues present LLMLingua, a token-level prompt compression method with a budget controller, and report compression of up to a factor of twenty with little loss on four benchmarks. Such a method belongs in the same evaluation as any other change, since compression that removes the one clause that decides a case is exactly the failure the class floors exist to catch.
The program varies the number of passages \(k\) in the prompt for both models on 20,000 requests. Recall is the share of requests whose evidence is in the top \(k\), a property of the retrieval list and independent of the model.
from world import *
reqs = make_requests(20000, seed=21)
F = 40.0
print("share of requests whose evidence is in the top-k list:")
print(" k " + " ".join(f"{k:5d}" for k in (1, 2, 3, 4, 5, 6, 8, 10, 12)))
print(" recall " + " ".join(f"{sum(r.gold_rank <= k for r in reqs)/len(reqs):5.3f}" for k in (1, 2, 3, 4, 5, 6, 8, 10, 12)))
for m in ("large", "small"):
print(f"\nmodel {m}")
print(f"{'k':>3s}{'input tok':>10s}{'success':>9s}{'spend':>7s}{'total':>7s}{'p50 ms':>8s}{'p95 ms':>8s}")
for k in (2, 3, 4, 5, 6, 8, 10, 12):
res = [call(m, r, k) for r in reqs]
s = sum(x[0] for x in res) / len(res); c = sum(x[1] for x in res) / len(res)
tin = PREFIX_TOK + 100 + k * PASSAGE_TOK
print(f"{k:3d}{tin:10d}{s:9.3f}{c:7.3f}{c+(1-s)*F:7.3f}{pct([x[2] for x in res],.5):8.0f}{pct([x[2] for x in res],.95):8.0f}")share of requests whose evidence is in the top-k list:
k 1 2 3 4 5 6 8 10 12
recall 0.591 0.809 0.891 0.921 0.932 0.937 0.938 0.939 0.939
model large
k input tok success spend total p50 ms p95 ms
2 1440 0.797 1.760 9.866 1580 2468
3 1660 0.855 1.980 7.794 1602 2503
4 1880 0.870 2.200 7.392 1624 2537
5 2100 0.871 2.420 7.588 1646 2572
6 2320 0.866 2.640 7.994 1668 2606
8 2760 0.854 3.080 8.916 1713 2674
10 3200 0.841 3.520 9.888 1757 2742
12 3640 0.828 3.960 10.850 1801 2811
model small
k input tok success spend total p50 ms p95 ms
2 1440 0.631 0.176 14.944 582 935
3 1660 0.677 0.198 13.136 591 948
4 1880 0.690 0.220 12.634 600 963
5 2100 0.689 0.242 12.692 609 977
6 2320 0.686 0.264 12.814 618 991
8 2760 0.676 0.308 13.266 635 1019
10 3200 0.665 0.352 13.734 653 1047
12 3640 0.655 0.396 14.208 671 1075Recall climbs from 0.591 at \(k=1\) to 0.921 at \(k=4\), then flattens at about 0.94, because the remaining 6% of requests never retrieve their evidence. For the large model, success rises from 0.797 at \(k=2\) to 0.871 at \(k=5\) and then declines, to 0.828 at \(k=12\). The decline is the toy’s built-in distraction penalty, so its shape is an assumption, whereas the flat recall is a consequence of the data. The total cost per task falls to its minimum at \(k=4\) (7.392 u), is 7.588 u at \(k=5\), and reaches 10.850 u at \(k=12\). Moving from eight passages to four cuts the large model’s spend from 3.080 u to 2.200 u, a 29% saving, and the toy’s quality does not fall. The small model’s curve has the same shape at a lower level.
The latency columns show a different effect. Each extra passage adds 220 tokens, about 22 ms at the median in this configuration, so moving from \(k=12\) to \(k=4\) removes about 180 ms from the large model’s median of roughly 1,800 ms. The large model’s output of 80 tokens at 14 ms each accounts for 1,120 ms of the median by itself. Context reduction therefore buys cost reliably and latency only in proportion to the share of time spent in prefill. When latency is the problem, the first questions are how many tokens the answer needs to be, how many calls are sequential, and which model answers, and context size comes after them. Reducing the output length, for example with a more compact schema, helps both: the output cap is also a latency lever, as the model API article noted.
Tail latency
The p95 and p99 of a latency distribution are what the slowest customers experience, and the percentile definition is the same as in the framing article. The Google SRE book makes the underlying point about averages: a mean can obscure the tail and changes in it. What a model application adds is a long list of components that each contribute to the tail: waiting for a rate limit, the provider’s queue, prefill proportional to input tokens, decode proportional to output tokens, validation and repair, retries, tool calls, and the second call of a cascade. Three properties of tails matter for optimization.
First, tails compound across calls that must all finish. If each of \(k\) independent calls misses its threshold with probability \(q\), the request misses with probability \(1-(1-q)^k\). A per-call p95 turns into a miss rate of 22.6% across five calls, and a per-call p99 into 4.9%. Independence is an optimistic assumption, since shared load and shared provider incidents correlate the calls. Sequential chains add durations, and parallel fan-out takes the maximum; Dean and Barroso’s “The Tail at Scale” is the standard treatment of how rare slowness in components becomes common slowness in the whole, and of the techniques that tolerate it.
Second, the techniques that tolerate a tail work only for some tail shapes. A hedged request, described by Dean and Barroso, sends a second copy of a request if the first has not completed after a delay, and uses whichever answer arrives first. It helps when the tail comes from a rare slow replica and the typical latency is short compared with the delay. A timeout with fallback bounds the worst case at the timeout plus the fallback’s own time. The program below applies both to the large model on 40,000 requests, with hedge delays set at several percentiles of the single-call latency. It bills both copies of a hedged request in full, which is the worst case for cost.
import random, statistics
from world import *
K = 5
reqs = make_requests(40000, seed=31)
base = [call("large", r, K) for r in reqs]
lat = [x[2] for x in base]; unit = statistics.fmean(x[1] for x in base)
TIMEOUT = 2500 # ms: give up on the large model, answer with the small one
def summarize(name, lats, cost, succ):
print(f"{name:34s} p50 {pct(lats,.5):5.0f} p95 {pct(lats,.95):5.0f} p99 {pct(lats,.99):5.0f} cost x{cost:5.3f} success {succ:.3f}")
summarize("single call, large", lat, 1.0, sum(x[0] for x in base) / len(base))
for q in (.5, .75, .90, .95): # hedge delay as a percentile of single-call latency
H = pct(lat, q); l2, c2 = [], 0.0
for r, (ok, c, l, _) in zip(reqs, base):
if l > H: # no answer by the hedge delay: send a second identical request, take the first to finish
_, c_b, l_b, _ = call("large", r, K, attempt=1)
l, c = min(l, H + l_b), c + c_b # both requests are billed in full (worst case)
l2.append(l); c2 += c
summarize(f"hedge after p{q*100:.0f} = {H:.0f} ms", l2, c2 / len(reqs) / unit, sum(x[0] for x in base) / len(base))
l3, c3, s3, nfb = [], 0.0, 0, 0
for r, (ok, c, l, _) in zip(reqs, base):
if l > TIMEOUT: # timeout, then fall back to the small model
ok_s, c_s, l_s, _ = call("small", r, K); ok, c, l = ok_s, c + c_s, TIMEOUT + l_s; nfb += 1
l3.append(l); c3 += c; s3 += ok
summarize(f"timeout {TIMEOUT} ms, then small model", l3, c3 / len(reqs) / unit, s3 / len(reqs))
print(f" requests that hit the timeout: {nfb/len(reqs):.3%}")
print("\nspread of the p99 estimate over 300 samples of size n from the same distribution:")
rng = random.Random(1)
for n in (200, 2000, 20000):
est = [pct(rng.sample(lat, n), .99) for _ in range(300)]
print(f" n = {n:5d}: mean {statistics.fmean(est):6.0f} ms sd {statistics.stdev(est):5.0f} ms")
print("\nfan-out: share of requests slower than the per-call threshold when k independent calls must all finish")
print(" per-call miss k=1 k=3 k=5 k=8")
for q in (.05, .01):
print(f" {q:8.0%} " + " ".join(f"{1-(1-q)**k:6.1%}" for k in (1, 3, 5, 8)))single call, large p50 1646 p95 2560 p99 3355 cost x1.000 success 0.875
hedge after p50 = 1646 ms p50 1646 p95 2558 p99 3092 cost x1.500 success 0.875
hedge after p75 = 1961 ms p50 1646 p95 2560 p99 3220 cost x1.250 success 0.875
hedge after p90 = 2307 ms p50 1646 p95 2560 p99 3342 cost x1.100 success 0.875
hedge after p95 = 2560 ms p50 1646 p95 2560 p99 3355 cost x1.050 success 0.875
timeout 2500 ms, then small model p50 1646 p95 2960 p99 3291 cost x1.006 success 0.865
requests that hit the timeout: 5.812%
spread of the p99 estimate over 300 samples of size n from the same distribution:
n = 200: mean 3681 ms sd 653 ms
n = 2000: mean 3393 ms sd 193 ms
n = 20000: mean 3359 ms sd 47 ms
fan-out: share of requests slower than the per-call threshold when k independent calls must all finish
per-call miss k=1 k=3 k=5 k=8
5% 5.0% 14.3% 22.6% 33.7%
1% 1.0% 3.0% 4.9% 7.7%Hedging gives little here. Hedging after the median reduces the p99 from 3,355 ms to 3,092 ms and costs 50% more; hedging after the 90th or 95th percentile leaves the p99 almost unchanged. The reason is in the shape of the distribution. A typical large-model call takes about 1,650 ms because generation dominates, so a second call started at 2,300 ms has little chance to finish before the first one that is merely slow. The slow path adds only about 1.2 s on average to a 1.6 s body, so the p99 is only twice the median. A distribution with a short body and a rare very slow component would respond differently, and the hedge is evaluated on the application’s own latency distribution, not adopted from a paper.
The timeout is more instructive, because it worsens the p95. With a timeout of 2,500 ms, the 5.8% of requests that exceed it are answered by the small model, which takes about 600 ms more, so those requests now finish near 3.1 s and the p95 moves from 2,560 ms to 2,960 ms. The p99 improves only from 3,355 ms to 3,291 ms, and success falls from 0.875 to 0.865 because the small model answers the slowest requests. A fallback has to satisfy a deadline in two parts: the timeout plus the fallback’s p95 must fit within the budget, or the fallback converts some requests that were slow into requests that are slower and wrong. The deadline-propagation rule from the model API article applies: each stage receives the remaining time and the fallback is only started if it can finish. The second lever on the p95 here is not a protocol at all but a smaller or shorter generation, as the decode share in the context section suggests.
Third, tails need samples to be measured. The last block of the output re-estimates the p99 from repeated samples of the same distribution. With 200 requests the estimate has a standard deviation of 653 ms, with 2,000 it is 193 ms, and with 20,000 it is 47 ms. A canary of a few hundred requests cannot distinguish a p99 that moved by 300 ms from noise, so early rollout stages guard on quantities that are visible at small scale, such as the timeout rate, the escalation rate, the error rate, and spend per request, and the p95 or p99 is compared at a later stage with enough traffic. The sample-size reasoning of the rollout article applies, and the output above shows how much more data a high percentile needs than a mean. Latency measured at the unloaded service of this toy is also a lower bound: under load, queueing adds to every component, and the capacity discussion in capacity, load shedding, and resilience testing covers how that is tested.
Evaluation budgets
Choosing among configurations is a measurement, and measurements cost money. Evaluating a configuration on a case spends the candidate’s own calls, the judge’s calls, and sometimes a person’s labeling time. With \(m\) configurations, \(n\) cases, and \(r\) runs per case, the evaluation spend is about \(m\,n\,r\,(\bar c + j)\), where \(\bar c\) is the configuration’s mean spend per case and \(j\) the cost of judging one answer. That number competes with production spend and with the time of whoever reads disagreements, so an evaluation plan is designed to spend where the decision is uncertain. Three habits keep it affordable.
- Screen cheaply, then spend on the survivors. Many candidates are clearly worse than the best on a small sample. Successive halving evaluates all candidates on a small number of cases, keeps the better half, and doubles the cases for the survivors. Jamieson and Talwalkar analyze this idea for best-arm identification and hyperparameter search, where it allocates more resources to promising settings.
- Reuse what does not change. Cases, retrieval results, and judge verdicts are keyed by their inputs and reused across configurations that share them. The reuse is keyed by release so that an evaluation of one configuration is never answered from another’s cache, for the reason given in the caching section.
- Use the cheapest valid instrument first. Deterministic checks run before a model judge, and a judge runs before a person, as in the layers of the reproducible evaluation article. A person’s time goes to the strata where the cheaper instruments disagree.
A fourth issue is statistical and not financial. When the best of \(m\) configurations is chosen by its score on one sample, that score is biased in the optimistic direction, because the winner was selected partly for being lucky on that sample. This is the winner’s curse. The size of the bias grows with \(m\) and shrinks with the number of cases, and the remedy is to confirm the winner on a sample that played no part in the selection. The program below builds twelve configurations from four cascade thresholds and three context sizes, evaluates them on a large reference sample that stands in for the truth, and compares three evaluation plans over 200 repeated draws of the development sample. The judge costs 0.3 u per case, an assumption.
import random, statistics
from configs import *
_, ref = load("ref", 20000, 41, 200000) # large reference sample: stands in for the true population values
dev_reqs, dev = load("dev", 4000, 42, 300000)
truth = {c: total(ref) for c in CONFIGS}
best = min(truth, key=truth.get)
print("population value of each configuration (total cost per task, u); lower is better")
for c in sorted(CONFIGS, key=truth.get)[:6]:
print(f" tau {c[0]:.1f}, k {c[1]:2d}: {truth:.3f}")
print(f" ... worst: tau {max(truth, key=truth.get)[0]:.1f}, k {max(truth, key=truth.get)[1]}: {max(truth.values()):.3f}")
per_case = lambda c, i: dev[i][1] + J # spend to evaluate one case: the candidate call(s) plus the judge
def strategy(sizes, keep, rng):
idx = rng.sample(range(len(dev_reqs)), sizes[-1]) # nested prefixes of one shuffled sample: later rounds extend earlier ones
alive, spent, done = list(CONFIGS), 0.0, 0
for n, kp in zip(sizes, keep):
for c in alive:
spent += sum(per_case(c, i) for i in idx[done:n])
done = n
est = {c: total([dev[i] for i in idx[:n]]) for c in alive}
alive = sorted(alive, key=est.get)[:kp]
return alive[0], est[alive[0]], spent
rng = random.Random(7)
plans = {"all 12 on 1,200 cases": ([1200], [1]), "all 12 on 150 cases": ([150], [1]),
"halving 150/300/600/1200": ([150, 300, 600, 1200], [6, 3, 2, 1])}
print(f"\n{'plan':26s}{'eval spend':>11s}{'picks best':>11s}{'mean regret':>12s}{'optimism (est. - truth)':>27s}")
spends = {}
for name, (sizes, keep) in plans.items():
hit, reg, spend, optim = 0, [], [], []
for _ in range(200):
pick, est, sp = strategy(sizes, keep, rng)
hit += pick == best; reg.append(truth[pick] - truth[best]); spend.append(sp); optim.append(est - truth[pick])
spends[name] = statistics.fmean(spend)
print(f"{name:26s}{statistics.fmean(spend):11.0f}{hit/200:11.2f}{statistics.fmean(reg):12.3f}{statistics.fmean(optim):27.3f}")
prod = sum(x[1] for x in ref[best]) / len(ref[best])
print(f"\nspend of the best configuration per production request: {prod:.3f} u")
print(f"the halving plan's evaluation spend equals that of {spends['halving 150/300/600/1200']/prod:,.0f} production requests")population value of each configuration (total cost per task, u); lower is better
tau 0.7, k 5: 6.877
tau 0.8, k 5: 7.112
tau 0.6, k 5: 7.162
tau 0.7, k 3: 7.450
tau 0.8, k 3: 7.615
tau 0.6, k 3: 7.760
... worst: tau 0.5, k 8: 9.197
plan eval spend picks best mean regret optimism (est. - truth)
all 12 on 1,200 cases 26129 0.82 0.090 0.007
all 12 on 150 cases 3266 0.34 0.338 -0.417
halving 150/300/600/1200 8849 0.74 0.141 -0.018
spend of the best configuration per production request: 1.701 u
the halving plan's evaluation spend equals that of 5,202 production requestsThe reference values show how the configurations differ: the best, a threshold of 0.7 with five passages, has a total cost of 6.877 u per task, and the next two are 7.112 u and 7.162 u. Evaluating all twelve on 1,200 cases costs 26,129 u of evaluation spend, finds the best configuration in 82% of the repetitions, and loses 0.090 u per task on average when it does not; its estimate of the chosen configuration is unbiased to within 0.007 u. Evaluating all twelve on 150 cases costs 3,266 u, finds the best only 34% of the time with a regret of 0.338 u, and reports for the configuration it picks a cost 0.417 u lower than the truth, which is the winner’s curse. Successive halving from 150 to 1,200 cases costs 8,849 u, finds the best in 74% of the repetitions, loses 0.141 u on average, and shows a small optimistic bias of 0.018 u, since the survivors were evaluated on 1,200 cases at the end. The halving plan spends a third of the full plan and gets most of its accuracy. Its evaluation spend equals the spend of 5,202 production requests of the chosen configuration, so a plan is judged against the volume that the savings will apply to.
Two cautions apply to reading the table. Picking exactly the best configuration is a stricter standard than the decision needs, because the two best configurations, at 6.877 u and 7.112 u, differ by 0.235 u while the best and the worst differ by 2.320 u, and the regret column is the more relevant one. And the quality of the plan depends on the spread of the true values, which in a real system is unknown. A practical rule is to state in advance a margin below which two configurations are treated as equivalent, stop evaluating a configuration once the paired interval of its difference from the leader excludes that margin, and spend the remaining budget on the leader and on the configurations that are within the margin.
Controlled comparisons
A comparison is controlled when the only thing that differs between the arms is the thing under test, and the measurement distinguishes a real difference from the noise of the sample. The chosen configuration in this article, a cascade at \(\tau=0.7\) with five passages, bundles two changes, routing and context reduction, and the baseline is the current release: the large model with eight passages. Three decisions structure the comparison.
First, the arms run on the same cases, so the comparison is paired. The variation from some requests being harder than others cancels in the per-case difference, as the reproducible-evaluation article showed, and the interval for the difference narrows accordingly. Second, the claim is stated as noninferiority for quality and as a difference for cost and latency, with margins written before the data arrives: the lower bound of the 95% interval of the success difference must be at least \(-0.02\) overall and per critical class, and the p95 may rise by at most 300 ms (a hypothetical latency criterion for this example). Third, cost and latency are compared by statistics that fit them: spend and total cost per task as means with paired intervals, and the p95 by resampling whole cases, so both arms of each resampled case stay together, and computing the percentile of each arm on every resample. The bootstrap does this without a formula for the variance of a percentile, and comparing models develops the paired comparison for model quality in general.
import random, statistics
from configs import *
CAND, MARGIN, N, REPS = (.7, 5), -0.02, 3000, 2000
reqs = make_requests(N, 51, first_id=500000)
base = [call("large", r, 8)[:3] for r in reqs] # current release: large model, 8 passages
cand = [run_config(CAND, r) for r in reqs] # candidate: cascade tau 0.7, 5 passages (chosen on the dev sample)
rng = random.Random(4)
def stat(idx):
b = [base[i] for i in idx]; c = [cand[i] for i in idx]
ds = sum(x[0] for x in c) / len(idx) - sum(x[0] for x in b) / len(idx)
spend = lambda o: sum(x[1] for x in o) / len(o)
dtot = total(c) - total(b)
return ds, dtot, pct([x[2] for x in c], .95) - pct([x[2] for x in b], .95), spend(c) / spend(b)
point = stat(range(N))
boots = [stat(rng.choices(range(N), k=N)) for _ in range(REPS)] # paired: the same resampled cases feed both arms
ci = lambda j: (sorted(b[j] for b in boots)[int(.025 * REPS)], sorted(b[j] for b in boots)[int(.975 * REPS)])
labels = ["delta success", "delta total cost/task (u)", "delta p95 latency (ms)", "spend ratio cand/base"]
for j, name in enumerate(labels):
lo, hi = ci(j); print(f"{name:27s} {point[j]:+9.3f} 95% CI [{lo:+.3f}, {hi:+.3f}]")
# an unpaired interval for the success difference, for comparison
ub = []
for _ in range(REPS):
a = rng.choices(base, k=N); b = rng.choices(cand, k=N)
ub.append(sum(x[0] for x in b) / N - sum(x[0] for x in a) / N)
ub.sort(); pb = sorted(b[0] for b in boots)
print(f"success difference, paired width {pb[int(.975*REPS)]-pb[int(.025*REPS)]:.3f} vs unpaired width {ub[int(.975*REPS)]-ub[int(.025*REPS)]:.3f}")
print(f"\n{'class':13s}{'n':>5s}{'base':>7s}{'cand':>7s}{'delta':>8s}{'95% CI':>18s}")
for c in CLASSES:
ix = [i for i, r in enumerate(reqs) if r.cls == c]
d = []
for _ in range(REPS):
s = rng.choices(ix, k=len(ix)); d.append(sum(cand[i][0] - base[i][0] for i in s) / len(ix))
d.sort()
print(f"{c:13s}{len(ix):5d}{sum(base[i][0] for i in ix)/len(ix):7.3f}{sum(cand[i][0] for i in ix)/len(ix):7.3f}"
f"{sum(cand[i][0]-base[i][0] for i in ix)/len(ix):+8.3f} [{d[int(.025*REPS)]:+.3f}, {d[int(.975*REPS)]:+.3f}]")
lo = ci(0)[0]
print(f"\nnoninferiority on success (lower bound >= {MARGIN}): {lo >= MARGIN}; p95 within +300 ms: {ci(2)[1] <= 300}")
# one change at a time, on the same 3,000 cases
arms = {"baseline: large, k=8": base,
"large, k=5 (context only)": [call("large", r, 5)[:3] for r in reqs],
"cascade 0.7, k=8 (routing only)": [run_config((.7, 8), r) for r in reqs],
"cascade 0.7, k=5 (both)": cand}
print(f"\n{'arm':32s}{'success':>8s}{'spend':>7s}{'p50':>6s}{'p95':>6s}{'total':>7s}")
for name, o in arms.items():
print(f"{name:32s}{sum(x[0] for x in o)/N:8.3f}{sum(x[1] for x in o)/N:7.3f}{pct([x[2] for x in o],.5):6.0f}{pct([x[2] for x in o],.95):6.0f}{total(o):7.3f}")delta success +0.011 95% CI [+0.006, +0.017]
delta total cost/task (u) -1.781 95% CI [-2.011, -1.561]
delta p95 latency (ms) +530.678 95% CI [+459.519, +565.936]
spend ratio cand/base +0.565 95% CI [+0.550, +0.578]
success difference, paired width 0.011 vs unpaired width 0.033
class n base cand delta 95% CI
standard 1613 0.889 0.900 +0.011 [+0.004, +0.019]
late 482 0.892 0.900 +0.008 [-0.004, +0.021]
final_sale 280 0.843 0.864 +0.021 [+0.007, +0.039]
missing_info 387 0.840 0.845 +0.005 [-0.010, +0.021]
conflict 238 0.697 0.710 +0.013 [-0.004, +0.034]
noninferiority on success (lower bound >= -0.02): True; p95 within +300 ms: False
arm success spend p50 p95 total
baseline: large, k=8 0.864 3.079 1706 2604 8.533
large, k=5 (context only) 0.880 2.419 1640 2509 7.206
cascade 0.7, k=8 (routing only) 0.858 2.236 2014 3249 7.902
cascade 0.7, k=5 (both) 0.875 1.739 1924 3134 6.752Quality is not worse. The success difference is +0.011 with a 95% interval of [+0.006, +0.017], so the noninferiority margin of \(-0.02\) is cleared with room to spare, and the interval for the hardest class, conflict, is [-0.004, +0.034] on 238 cases. The spend ratio is 0.565, and total cost per task falls by 1.78 u with an interval of [-2.01, -1.56]. The p95 rises by 531 ms, with an interval of [+460, +566], and the latency criterion of at most 300 ms fails. The paired interval for the success difference has a width of 0.011, against 0.033 for an unpaired comparison on the same 3,000 cases. Since interval width shrinks with the square root of the sample size, an unpaired comparison needs about nine times as many cases to reach the same precision.
The decision does not have to be all or nothing, and the last table shows why the bundle is split before the conclusion is drawn. Changing only the context, from eight passages to five on the large model, improves everything at once: success rises from 0.864 to 0.880, spend falls 21% from 3.079 u to 2.419 u, and the p95 falls from 2,604 ms to 2,509 ms. Changing only the routing cuts spend by 27% and lowers success slightly (0.858), but it pushes the p50 from 1,706 ms to 2,014 ms and the p95 from 2,604 ms to 3,249 ms. The two together reach the lowest total cost per task (6.752 u) and keep most of the routing’s latency cost. The context change can ship on its own. Whether the routing is adopted depends on whether the product can afford 500 to 650 ms more at the p95 for a 27% saving, and a cascade that escalates fewer requests, or that applies only to classes with a loose deadline, is a different candidate that has to be evaluated on its own. The decision belongs in the record of the framing article together with the margins, the sample, and the date.
Moving from the offline comparison to the rollout
Offline cost and latency numbers are conditional on the cache state and on the load of the test, and neither is the production state. The stages of the online-evaluation article carry the comparison forward, with cost and latency added to the guardrails of each stage.
| Stage | What it can confirm for quality, latency, and cost | Guardrails to add |
|---|---|---|
| Offline gate | Paired quality against the class floors; per-case cost and latency without load | Cost per task and p95 from the paired comparison |
| Shadow | Spend, token counts, escalation rate, and latency on live inputs; disagreement with the current release | Escalation rate within its expected range; cached-token share against the estimate; no actions taken |
| Canary | Gross breakage; timeout and error rates; spend per request | Timeout rate, error rate, escalation rate, spend per request; p99 is too noisy at this size |
| Ramp | The outcome difference against a concurrent control; the p95 at enough traffic; interference through shared caches and capacity | Sample ratio check; p95 by arm; capacity headroom on the large model |
| Full with holdback | Matured cost per successful task; drift in escalation and hit rates | Cost per successful task on matured cohorts; alerts on price or model changes |
Two of these deserve a sentence each. Shared capacity makes the arms interfere: a cascade that sends 61% of its traffic to the large model uses capacity that the control arm also needs, so the control’s latency can rise when the candidate’s exposure grows, and the comparison is checked for that. And the matured cost per successful task is the number that confirms the saving, because it counts failed tasks and only completed outcomes; the spend per request that the canary reads is the early indicator.
Where optimization stops
An optimization is a trade measured on a sample, and it holds for the traffic, prices, models, and load that the sample represented. Prices change, which moves the break-even \(F^{*}\) and the best threshold. A model update changes the check’s calibration and the router’s accuracy, so the thresholds are re-fitted and the comparison is repeated, using the same paired set. A shift in the request mix changes the per-class gaps and therefore the value of routing. Cost and latency measured offline do not include queueing under load. And a metric that is optimized becomes a target: if the quality measure misses a kind of failure, a cheaper configuration can introduce that failure and pass every gate, which is why the audit samples and the matured outcomes of the online-evaluation article stay in place after the optimization ships.
References
- Chen, L., Zaharia, M., and Zou, J. “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.” arXiv:2305.05176.
- Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., and Stoica, I. “RouteLLM: Learning to Route LLMs with Preference Data.” arXiv:2406.18665.
- Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. “LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.” EMNLP 2023, arXiv:2310.05736.
- Dean, J. and Barroso, L. A. “The Tail at Scale.” Communications of the ACM 56(2), 2013, pp. 74-80.
- Jamieson, K. and Talwalkar, A. “Non-stochastic Best Arm Identification and Hyperparameter Optimization.” arXiv:1502.07943.
- Anthropic. “Prompt caching.” Claude API documentation (cache breakpoints, lifetimes, pricing structure, and usage fields).
- Beyer, B., Jones, C., Petoff, J., and Murphy, N. R. (eds.). “Service Level Objectives.” In Site Reliability Engineering, O’Reilly, 2016; available at sre.google/sre-book/service-level-objectives/.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.