AI Service Objectives and Reliability Engineering
Consider a hypothetical support assistant whose dashboard is green. Availability is 99.95 percent, the 95th-percentile latency is inside its target, and no alert has fired in three weeks. Meanwhile the support team reports that since a Tuesday release the assistant has been quoting the wrong refund window to about one customer in ten. Every one of those responses was an HTTP 200 containing well-formed text, delivered quickly.
Nothing in the dashboard is wrong. It measures what it was built to measure: whether the service answered, and how fast. A generative AI service can fail in a way that classical reliability engineering does not count, because the service worked and the answer did not. This post covers how to set service level objectives for such a system. It handles latency for responses that stream and vary in length, availability when a request chains through retrieval, a model provider, and tools, error budgets and burn-rate alerting, and the harder indicator of whether the task was done. It then covers degradation and which signals reveal an incident that error rates cannot.
It builds on Observing AI Systems Without Exposing Sensitive Data, which defines the content-free telemetry (durations, token counts, retrieval counts, tool outcomes, and numeric scores computed elsewhere) that the indicators below are built from. The reliability terms are defined as they appear; the only prior knowledge assumed is the idea of a request moving through a distributed service.
Indicators, objectives, and budgets
A service level indicator (SLI) is a measurement of how well the service is doing, and the most useful form is a ratio: good events divided by valid events. Google’s Site Reliability Workbook recommends this structure because it runs from 0 percent to 100 percent and makes the rest of the arithmetic uniform. What counts as an event, what counts as valid, and what counts as good are the whole content of the definition; most arguments about whether a service met its target are arguments about those three choices. See SLI (Service Level Indicator) for a fuller treatment of the definition.
A service level objective (SLO) is a target for an SLI over a window: for example, at least 99.9 percent of valid requests succeed over 30 days (SLO). The error budget is what the target leaves over. With a target \( s \) and \( N \) valid events in the window, the allowed number of bad events is \( (1 – s) \cdot N \). At a 99.9 percent target the budget is 0.1 percent of events. If traffic were steady and all failures were complete outages, that is 43.2 minutes of full outage in 30 days (30 days is 43,200 minutes, and 0.1 percent of that is 43.2). The budget exists so that releases and experiments can spend some of it. Google’s SRE book describes the mechanism as follows: as long as measured reliability is above the objective, meaning budget remains, new releases can go out (Error Budget).
For AI services this framework is the right structure and needs three adaptations. The latency indicator must account for output that streams and varies in length. The availability indicator must account for a chain of dependencies, some of which fail partially. And a third indicator, task success, must capture quality, which has no equivalent in the classical menu except through the correctness and quality entries the workbook lists.
Latency for streamed, variable-length output
A conventional latency SLI says that a request is good if it completes within some threshold. For a language model that streams its answer, that single number mixes two different experiences. The wait before anything appears is the time to first token (TTFT), which depends on queueing, prompt length, and prefill. After that, the text arrives at a decode rate, in tokens per second or milliseconds per token, which depends on the model and on how loaded the server is. Total time is roughly TTFT plus output length times time per token, so a long answer is slow by construction even when the system is healthy. OpenTelemetry’s GenAI metric conventions reflect this split with separate instruments for time to first token and time per output token on the server side, and time to first chunk on the client side; all are in Development status.
The experiment generates 200,000 synthetic requests with a long-tailed output length (median 200 tokens), a log-normally distributed TTFT (median 0.45 s), and a per-request decode time of about 22 ms per token. The model is a simulation of these distributions, not a measurement of any real model. Two SLIs are compared. The flat SLI calls a request good if its total time is at most 15 seconds. The split SLI calls it good if TTFT is at most 1 second and decode speed is at most 32 ms per token. Both thresholds are arbitrary choices for the exercise.
flat_good = total <= 15.0
scaled_good = (ttft <= 1.0) & (per_tok <= 0.032)
print(f"\nflat SLI (total <= 15 s) good = {flat_good.mean()*100:.2f}%")
print(f"split SLI (TTFT <= 1 s and decode <= 32 ms/token) good = {scaled_good.mean()*100:.2f}%")requests: 200,000
output tokens p50=200 p95=747 p99=1293
TTFT (s) mean=0.48 p50=0.45 p95=0.80 p99=1.02
total (s) mean=6.58 p50=4.87 p95=17.27 p99=29.84
flat SLI (total <= 15 s) good = 92.84%
split SLI (TTFT <= 1 s and decode <= 32 ms/token) good = 98.28%
output tokens share of traffic flat: bad % split: bad %
0-99 19.3% 0.00% 1.75%
100-199 30.6% 0.00% 1.67%
200-399 30.6% 0.00% 1.73%
400-799 15.2% 20.11% 1.74%
800-2000 4.2% 98.29% 1.86%
after a 25% decode slowdown on 20% of requests:
flat SLI good 92.84% -> 91.91% (change -0.92 points)
split SLI good 98.28% -> 95.28% (change -2.99 points)
after answers become 30% longer, with no change to the system:
flat SLI good 92.84% -> 87.36% (change -5.47 points)
split SLI good 98.28% -> 98.28% (change +0.00 points)The first lines show why averages mislead: the mean total time is 6.58 s while the 99th percentile is 29.84 s. The SRE book makes the same point with its own example, in which a typical request takes 50 ms and 5 percent of requests are twenty times slower, and recommends percentiles over means. See Percentile for how percentiles are computed and why they cannot be averaged across servers.
The table shows the problem with the flat SLI. Its failures are almost entirely long answers: 0 percent of answers under 400 tokens fail it, and 98.29 percent of answers over 800 tokens do. The flat SLI is therefore partly a measure of what users asked for. In the last scenario, answers become 30 percent longer while the system is untouched: the flat SLI drops 5.47 points and the split SLI does not move. Conversely, when 20 percent of requests are slowed by 25 percent in decode, the split SLI falls 2.99 points while the flat SLI falls 0.92, because most of the slowed requests were short enough to stay under the threshold. The split SLI separates the two causes: it responds to a change in the system and ignores a change in the workload.
A practical definition therefore has these parts. Measure TTFT for streaming responses and use total time for non-streaming ones, since a client that waits for the whole body has only one number. Express decode speed per output token, or scale the total-time threshold by the output length. Report the distribution by percentile. Segment by model and by request class, so a slow large model does not get averaged away by a fast small one. If requests have materially different expected cost (a short classification and a long document summary), give them different objectives. Finally, keep the boundary explicit: client-side TTFT includes network and gateway time that server-side TTFT does not, and the two are not interchangeable. Sequential steps in a path add as durations but their percentiles do not add: if two steps each take 100 ms on 99 of 100 requests and 1,000 ms on one request, with the slow cases on different requests, each step has a p99 of 100 ms while the path has a p99 of 1,100 ms. A per-step latency allocation has to be checked against the end-to-end distribution.
Availability across a chain of dependencies
What counts as a good request
Availability is the fraction of valid requests that succeed. The words “valid” and “succeed” need AI-specific rulings.
- Valid events. Exclude requests the service was right to reject: malformed input, authentication failures, and rate limits applied by design because a tenant exceeded its quota. Keep requests that failed because the platform or a provider ran out of capacity. A 429 can mean either, so tag the cause at the gateway.
- Success at the application boundary. A stream that begins with a 200 status and then breaks midway is a failure, though the HTTP status says otherwise. Count completion of the response, not the first byte.
- Refusals. A policy refusal of an unsafe request is the system working. A refusal of a legitimate request is a failure of quality. Availability does not distinguish them, which is one reason the task-success indicator below is needed.
- Partial success. An answer assembled without a tool that failed is neither a clean success nor a failure. Treat it as a separate state, as the degradation section does.
For distributed services, Google’s SRE book prefers a request success rate over a time-based uptime figure, because a service that is partly available worldwide is not usefully described by whether it was up (Availability).
Serial dependencies
A request in the support assistant needs the gateway, the retrieval service, a model provider, and, for some requests, a tool. If any required component fails and nothing masks the failure, the request fails. When failures are independent, availabilities multiply. If a request uses component \( i \) with probability \( u_i \) and that component has availability \( a_i \), the overall availability is approximately \[ A \approx \prod_i a_i^{\,u_i}. \] The product is lower than every factor, and it is dominated by the weakest component. The exponent \( u_i \) accounts for components that only some requests use. Independence is an assumption: a shared network path or a common upstream can make failures arrive together, so this is closer to a best case than an estimate.
The experiment assigns the four components availabilities of 99.95, 99.9, 99.5, and 99.8 percent (assumed for the exercise), with the tool needed by 70 percent of requests, and then simulates a model-provider stage over 30 days at one request per second. The provider’s 0.5 percent unavailability is split into 60 percent short, independent request failures and 40 percent contiguous 20-minute outages. Four configurations are run: a single provider, one retry, a second provider as fallback, and both. The fallback provider is modeled with a 0.5 percent independent per-request failure rate, and a last configuration makes a quarter of the primary’s outage windows also take the fallback down.
def sim(split_transient, outage_minutes_total, outage_len, p_fallback=None, retry=False, shared=0.0):
"""Return a boolean array of request success for the model provider stage."""
p_trans = (1 - 0.9950) * split_transient
p_out_frac = (1 - 0.9950) * (1 - split_transient)
# outage windows
n_win = max(1, int(round(MIN * p_out_frac / outage_len)))
down = np.zeros(MIN, bool)
starts = rng.choice(MIN - outage_len, n_win, replace=False)
for s in starts: down[s:s + outage_len] = True
down_req = np.repeat(down, REQ_PER_MIN)
trans = rng.random(N) < p_trans
ok = ~(down_req | trans)
if retry: # one retry: independent for transient, fails again in an outage
retry_ok = ~(down_req | (rng.random(N) < p_trans))
ok = ok | (trans & ~down_req & retry_ok)
if p_fallback is not None: # second provider, independent failures
fb_ok = rng.random(N) > p_fallback
if shared > 0: # a fraction of primary outage windows also take the fallback down
fb_down = np.zeros(MIN, bool)
for s in starts[: int(len(starts) * shared)]: fb_down[s:s + outage_len] = True
fb_ok &= ~np.repeat(fb_down, REQ_PER_MIN)
ok = ok | (~ok & fb_ok)
return okcomponent availabilities: {'gateway': 0.9995, 'retrieval': 0.999, 'model provider': 0.995, 'tool': 0.998}
serial product, tool needed by 70% of requests: 0.99212 -> 341 minutes of unavailability per 30 days
error budget for a 99.9% SLO over 30 days: 43.2 minutes of full outage
model-provider stage only, provider unavailability 0.50% (60% transient, 40% in 20-minute outages)
configuration availability minutes-equivalent down share of 99.9% budget
single provider 99.521% 206.9 479%
retry once 99.814% 80.4 186%
fallback provider 99.997% 1.3 3%
retry + fallback 99.999% 0.4 1%
... 25% shared outages 99.953% 20.3 47%Several conclusions follow. The serial product is 99.212 percent, which is 341 minutes of unavailability in 30 days against a 99.9 percent budget of 43.2 minutes: with those components and no mitigation, a 99.9 percent objective cannot be met, whatever the engineering quality of the rest of the system. The provider stage alone, at 99.521 percent in the simulation, consumes 479 percent of the budget.
A retry helps with the transient failures (the budget share falls from 479 to 186 percent) and does nothing for an outage, since the retry lands in the same outage. A second provider helps with both, bringing the stage to 99.997 percent, but only to the extent the failures are independent. When a quarter of outages are shared, because both providers sit behind the same region or the same cloud, the result is 99.953 percent, which uses 47 percent of the budget instead of 1 percent. The fallback provider’s own failures here are independent per request, which flatters the numbers; the exercise isolates the effect of correlation, not the absolute figure. The lesson for design is to ask what the primary and the fallback share before counting the fallback as redundancy. Retries also deserve a budget of their own: a retry against an overloaded provider adds load to the provider that is failing, a point the gateway post makes in the context of provider failover (see Model Gateways: Routing, Quotas, and Provider Failover).
Dependencies also change the rule for ownership. The SRE workbook notes a choice when an upstream dependency causes a missed objective: exempt the team from the budget policy because its own code was not at fault, or apply the policy regardless, and it recommends applying it, because users experience the service as a whole. For a platform team, that argues for writing the dependency’s availability into the plan up front instead of absorbing it as a surprise.
Error budgets and burn-rate alerts
An alert that fires when the error rate crosses the SLO’s allowed rate sounds natural and performs badly. The SRE workbook works through the arithmetic for a 99.9 percent objective over 30 days. A 0.1 percent error rate sustained for 10 minutes consumes only about 0.02 percent of the monthly budget, yet a rule that pages on it can fire up to 144 times a day while the objective is still comfortably met. Lengthening the alert window to compensate creates a different failure: a complete outage trips the alert, and the alert keeps firing for hours after the outage ends. See Alert Fatigue for what noisy paging does to responders.
The workbook’s replacement is built on burn rate, the observed error ratio divided by the allowed ratio \( 1 – s \). A burn rate of 1 means the budget lasts exactly the window; a burn rate of 14.4 on a 30-day window consumes 2 percent of the budget in one hour (Burn Rate). For a 99.9 percent objective, the workbook recommends combining a long window with a short one, so an alert needs both to be over the threshold, which resets quickly after the problem is fixed:
| Severity | Long window | Short window | Burn rate | Budget consumed if it persists |
|---|---|---|---|---|
| Page | 1 hour | 5 minutes | 14.4 | 2% |
| Page | 6 hours | 30 minutes | 6 | 5% |
| Ticket | 3 days | 6 hours | 1 | 10% |
The next experiment compares four alert policies on a 30-day synthetic series of about 8.6 million requests with a diurnal traffic pattern (100 to 300 requests per minute) and a 0.03 percent background error rate. Four incidents are injected: a ten-minute blip at 1 percent errors, a ten-minute complete outage, an eight-hour partial failure at 5 percent errors, and a three-day slow burn at 0.4 percent errors. The policies are the naive rule (error ratio above 0.1 percent over 10 minutes), a single-window rule (burn rate 14.4 over one hour), the workbook’s two paging rules with short windows, and those plus the 3-day ticket rule.
R5, R10, R30, R60, R360, R4320 = (ratio(w) for w in (5, 10, 30, 60, 360, 4320))
R6h = R360
policies = {
"A: error ratio > 0.1% over 10 min": (R10 > BUDGET),
"B: burn rate >= 14.4 over 1 h": (R60 >= 14.4 * BUDGET),
"C: multiwindow (page rules only)": ((R60 >= 14.4 * BUDGET) & (R5 >= 14.4 * BUDGET)) | ((R360 >= 6 * BUDGET) & (R30 >= 6 * BUDGET)),
"D: multiwindow + 3 d ticket rule": ((R60 >= 14.4 * BUDGET) & (R5 >= 14.4 * BUDGET)) | ((R360 >= 6 * BUDGET) & (R30 >= 6 * BUDGET)) | ((R4320 >= BUDGET) & (R360 >= BUDGET)),
}requests over 30 days: 8,646,348; errors: 13,118; 30-day error budget: 8,646 errors
overall availability if nothing is done: 99.848%
incident total: blip: 1% errors for 10 min 41 errors = 0.5% of the 30-day budget
incident total: full outage, 10 min 1,808 errors = 20.9% of the 30-day budget
incident total: partial: 5% errors, 8 h 5,401 errors = 62.5% of the 30-day budget
incident total: slow burn: 0.4% errors, 3 d 3,502 errors = 40.5% of the 30-day budget
burn rate = (error ratio) / (1 - SLO); at burn rate 1 the budget lasts exactly the 30-day window
A: error ratio > 0.1% over 10 min: 513 alert episodes, 464 of them outside any incident
blip: 1% errors for 10 min first alert 0 min after start; errors so far = 0.1% of the 30-day budget
full outage, 10 min first alert 0 min after start; errors so far = 1.9% of the 30-day budget
partial: 5% errors, 8 h first alert 0 min after start; errors so far = 0.2% of the 30-day budget
slow burn: 0.4% errors, 3 d first alert 0 min after start; errors so far = 0.0% of the 30-day budget
B: burn rate >= 14.4 over 1 h: 2 alert episodes, 0 of them outside any incident
blip: 1% errors for 10 min not detected
full outage, 10 min first alert 0 min after start; errors so far = 1.9% of the 30-day budget
partial: 5% errors, 8 h first alert 17 min after start; errors so far = 3.0% of the 30-day budget
slow burn: 0.4% errors, 3 d not detected
C: multiwindow (page rules only): 2 alert episodes, 0 of them outside any incident
blip: 1% errors for 10 min not detected
full outage, 10 min first alert 0 min after start; errors so far = 1.9% of the 30-day budget
partial: 5% errors, 8 h first alert 17 min after start; errors so far = 3.0% of the 30-day budget
slow burn: 0.4% errors, 3 d not detected
D: multiwindow + 3 d ticket rule: 3 alert episodes, 0 of them outside any incident
blip: 1% errors for 10 min not detected
full outage, 10 min first alert 0 min after start; errors so far = 1.9% of the 30-day budget
partial: 5% errors, 8 h first alert 17 min after start; errors so far = 3.0% of the 30-day budget
slow burn: 0.4% errors, 3 d first alert 765 min after start; errors so far = 7.5% of the 30-day budgetThe naive rule produced 513 alert episodes, 464 of them outside any injected incident: the random fluctuation of a 0.03 percent background rate over 10-minute windows of about 2,000 requests crosses a 0.1 percent threshold regularly. The burn-rate rules had no alerts outside incidents. The ten-minute blip, which costs 0.5 percent of the month’s budget in total, correctly pages nobody. The outage pages immediately, after 1.9 percent of the budget has gone. The eight-hour partial failure pages after 17 minutes with 3 percent gone. The slow burn at 0.4 percent, a burn rate of 4, is invisible to the paging rules and is caught only by the 3-day ticket rule, after about 12.75 hours with 7.5 percent of the budget spent. A burn rate of 4 gives about a week before the budget is gone, which is the right urgency for a ticket and not for a page.
The relevant limit for AI services is what these alerts count. Burn rate over an error-ratio SLI detects hard failures. The Tuesday regression in the opening scenario produced none.
A task-success indicator
Defining the ratio
A task-success SLI asks what fraction of valid tasks the system completed correctly. The difficulty is that every part of the ratio needs a decision.
- The unit. A request, a conversation, and a task are different grains. A user may take four turns to get a refund processed; counting each turn dilutes the failures, counting the conversation matches the user’s goal. Choose the grain that corresponds to what the user is trying to do, and define when a task starts and ends.
- Valid. Exclude requests outside the assistant’s scope and tasks the user abandoned for reasons unrelated to the system. Record the exclusion rate, since a rising exclusion rate can hide a quality problem.
- Good. There are three ways to judge it, with different costs. A deterministic check on a structured outcome works when the task has one: the refund created has the amount and order that the user asked for. Human or model-based review of sampled conversations covers open-ended tasks, at a cost per judgment and with a judge error rate. Behavioral proxies (the user did not contact support again within seven days, the refund was not reversed) are cheap and automatic, and arrive late and are confounded by other causes.
- The sample. Judging every task is rarely affordable, so the SLI is an estimate with a confidence interval, and the interval is part of the definition.
Content-free telemetry connects to this directly. The judge runs in an environment that may read content, and it writes back a numeric verdict per sampled request, as described in the previous post. The SLI dashboard handles scores and counts, not conversations.
An experiment: a regression the error rate cannot see
The simulation runs 30 days at 5,000 requests per day. Hard failures (5xx responses and timeouts) occur with probability 0.08 percent throughout. True task success is 90 percent until day 15 and 82 percent from day 16, when a prompt template change ships; the numbers are assumptions chosen for the exercise. A judge reviews 100 requests per day, and its verdict is wrong 5 percent of the time in either direction, so a true rate \( p \) is observed as \( 0.05 + 0.9\,p \): 86 percent before the change and 78.8 percent after. Three days of samples are pooled into each row, and the interval is a 95 percent Wilson score interval for a proportion.
days error-rate SLI true task success sampled estimate (n=300, 95% CI)
1-3 99.93% 90.0% 87.3% [83.1, 90.6]
4-6 99.91% 90.1% 87.0% [82.7, 90.3]
7-9 99.92% 89.8% 82.7% [78.0, 86.5]
10-12 99.94% 89.7% 86.3% [82.0, 89.8]
13-15 99.96% 89.9% 88.0% [83.8, 91.2]
16-18 99.93% 81.9% 80.0% [75.1, 84.1]
19-21 99.93% 81.5% 82.7% [78.0, 86.5]
22-24 99.89% 81.6% 79.0% [74.0, 83.2]
25-27 99.88% 82.2% 80.0% [75.1, 84.1]
28-30 99.96% 81.9% 79.0% [74.0, 83.2]The error-rate SLI stays between 99.88 and 99.96 percent across the whole month. True task success drops from about 90 percent to about 82 percent. The sampled estimate shows the drop, but a single three-day window is noisy: the 7-9 row dips to 82.7 percent before the change, and its interval overlaps the post-change intervals. The judge’s own error also shifts the level, since the sampled baseline reads about 86 percent when the true figure is 90. So the SLI is only useful when its target is set against the judge’s measured baseline and the judge is itself calibrated against human labels.
How many samples are needed depends on the size of the drop one wants to catch quickly. The second experiment repeats the scenario 400 times for each combination of daily sample size and drop size. The rule fires when the upper bound of the pooled three-day interval is below 83 percent:
THRESH = 0.83
def trial(per_day_samples, drop_after):
ver = np.zeros(DAYS, int)
n = per_day_samples
for day in range(DAYS):
p = TASK_BEFORE if day < CHANGE_DAY else drop_after
truth = rng.random(n) < p
verdict = np.where(rng.random(n) < JUDGE_FLIP, ~truth, truth)
ver[day] = verdict.sum()
first = None; false_alert = False
for day in range(2, DAYS):
k = ver[day - 2: day + 1].sum(); nn = 3 * n
if wilson(int(k), nn)[1] < THRESH:
if day < CHANGE_DAY: false_alert = True
elif first is None: first = day
return first, false_alertsamples judged per day vs. detection (alert if the upper bound of the 3-day pooled success rate < 83%)
samples/day true drop detected within 14 d median delay (days) false alert before change
20 8 pts 81% 6.0 7.5%
50 8 pts 90% 5.0 1.5%
100 8 pts 99% 3.0 0.2%
300 8 pts 100% 3.0 0.0%
1000 8 pts 100% 2.0 0.0%
20 4 pts 41% 7.0 7.5%
50 4 pts 30% 8.0 1.0%
100 4 pts 39% 8.0 0.2%
300 4 pts 45% 8.0 0.0%
1000 4 pts 72% 7.0 0.0%For an 8-point true drop, 100 samples per day detects the change within 14 days in 99 percent of trials, with a median delay of three days, and a false-alert rate of 0.2 percent in the 15 days before the change. At 20 samples per day, delay and detection get worse (81 percent detected, median delay six days) and 7.5 percent of trials raise a false alert before anything changed. A 4-point drop is detected in at most 45 percent of trials at up to 300 samples per day and in 72 percent at 1,000, because that drop leaves the judged rate around 82.4 percent and the rule fires only when the upper bound is below 83 percent. This mismatch between the threshold and the shift to detect is a design error that no sample size repairs. Choose the threshold from the smallest drop that matters to users, then size the sample to detect it in an acceptable time.
Task success also fits the error-budget framework with adjustments. The budget in “bad tasks” is \( (1 – s) \cdot N \) as before, but both the numerator and the denominator are estimated from a sample, so burn-rate alerts on this SLI should use longer windows than error-rate alerts and should account for the interval. Page on the deterministic structured-outcome checks, where an error is a fact, and open tickets on the sampled judge, where it is an estimate. The offline evaluation from Prompt and Model Promotion with Evaluation Gates is the pre-release version of the same question; this SLI is the post-release version, and its sampled cases are good candidates for the next evaluation set.
Degradation as a designed state
When a dependency fails, an AI service has more options than failing the request. It can route to a smaller or different model (Model Fallback), serve a cached answer to a repeated question, drop the optional parts of the answer (skip the personalized lookup), queue the work for later, or return a limited fixed response. Each is a deliberate reduction in what the service provides, known as graceful degradation, and each is accounted for by an SLI.
Google’s workbook has a name for this: a quality indicator measures the proportion of responses served in an undegraded state. Availability counts whether any answer came back. The undegraded fraction counts whether the answer was the full one. Task success counts whether the answer worked. The three diverge in an outage, and the simulation below shows how. One million requests, 1 percent of which arrive during a primary-model outage; primary task success is 90 percent. Each mode has a share of outage-time requests it can serve and a task-success rate for what it serves. Those rates are assumptions for the exercise, not measured quality.
requests: 1,000,000; arriving during a primary-model outage: 1.00%; primary task success 90%
mode availability undegraded task success task success | outage
fail closed (error) 99.00% 99.00% 89.15% 0.0%
smaller fallback model 100.00% 99.00% 89.93% 78.0%
cached answer for repeat questions 99.30% 99.00% 89.32% 17.9%
answer without the tool (no lookup) 100.00% 99.00% 89.59% 44.9%Failing closed gives 99.00 percent availability. A smaller fallback model brings availability to 100 percent and overall task success rises from 89.15 to 89.93 percent, close to the 90 percent ceiling, though the requests served during the outage succeed at 78 percent rather than 90. A cache that can answer 30 percent of requests leaves availability at 99.30 percent. Answering without the tool reaches 100 percent availability, but the requests served during the outage succeed only 44.9 percent of the time. In every mode the undegraded fraction is 99.00 percent, since the outage requests are by definition not undegraded.
The point of the comparison is that availability alone rewards the wrong behavior. A system that always returns something, however poor, scores 100 percent. If availability is the only SLI, the cheapest way to meet the objective is to serve the weakest fallback for every failure. Reporting the undegraded fraction beside it, and task success beside that, keeps the trade-off visible. Four rules keep degradation from becoming a source of harm:
- Evaluate each degraded path against the task before admitting traffic to it, using the same evaluation sets as release gates. A fallback model can differ in refusals, structure, and tool support as well as accuracy.
- Decide which requests may degrade. A balance lookup can fall back to a cached value with a visible timestamp; a refund approval should not fall back to a model that has not been evaluated for it.
- Tell the user when the answer is degraded, in the product or in the response, so a limited answer is not mistaken for the normal one.
- Check what the fallback path inherits. A cached answer for one tenant must not reach another tenant, and a cheaper fallback model must meet the same data-handling constraints as the primary. The next post returns to that.
Incident signals: what shows up before the error rate
An AI incident often begins as a change in distribution, not in error counts. The table lists signals that can be computed from content-free telemetry, with what each reveals and what it misses.
| Signal | Detects | Misses |
|---|---|---|
| Error ratio and burn rate | Hard failures, outages, provider errors | Wrong answers delivered successfully |
| TTFT and per-token decode time by model | Capacity problems, a slow replica pool, provider slowdowns | Quality changes; slowdowns hidden by a flat total-time threshold |
| Retrieval returned-count and top-score distributions | Index problems, embedding or ingestion changes, empty retrieval | Retrieval that returns plausible but wrong documents |
| Tool error and retry rates; tool latency | A failing downstream dependency, schema drift in a tool | A tool that succeeds with the wrong arguments |
| Finish-reason mix and output-token distribution | Truncation, runaway generation, prompt changes that lengthen output | Short but wrong answers |
| Fallback and cache-hit rates | A primary path degrading, a dependency outage, cache poisoning or misconfiguration | Quality of the fallback itself |
| Refusal rate | Over-blocking after a safety or policy change | Under-blocking |
| Sampled task-success score | Quality regression after a prompt, model, or data change | Small shifts in a small sample; delay |
| User feedback, retries, escalations | Problems the user notices | Problems the user does not notice |
Two practices improve all of these. First, attach the version of the prompt template, the model, the retrieval index, and the tool schema to every request, so any shift in a signal can be compared with the timeline of changes. A quality drop that starts at a deploy boundary, or at a provider’s silent model update, is far easier to diagnose than one with no timeline. Second, run synthetic probes: scheduled requests with known expected outcomes that go through the production path. Probes give a task-success estimate that does not depend on user traffic, and, as the previous post notes, they can be recorded in full because they contain no customer data.
A reasonable starting set of objectives for the support assistant would combine the pieces above: a TTFT and decode-speed SLI per model class, a request-based availability SLI at the application boundary that counts truncated streams as failures, a task-success SLI from structured checks and sampled review with its interval reported, and the undegraded fraction as a companion to availability. Page on burn rate for the first two and on deterministic failures in the third, open tickets for slow burn and sampled-quality drift, and review degraded-mode usage after any incident.
Reliability engineering treats failure as something the world does to the system. The next post takes up the case in which failure is induced on purpose: untrusted content steering the model, one tenant reaching another’s data through a shared index or cache, and the controls and tests that bound those risks. See AI Platform Security and Tenant Isolation.
References
- Google SRE book, Service Level Objectives
- Google SRE book, Embracing Risk
- Google SRE workbook, Implementing SLOs
- Google SRE workbook, Alerting on SLOs
- OpenTelemetry, Semantic conventions for generative AI (metrics and spans)
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.