SLOs, Alerting, and Error Budgets

Suppose a platform team runs a checkout service and has a dashboard with forty charts and thirty alert rules. CPU on any instance above 80 percent pages someone. A single failed health check pages someone. A five-minute error rate above 0.1 percent pages someone. On a bad week the on-call engineer is woken up repeatedly, and on the week that real users cannot pay, the page looks exactly like all the others. Every alert is technically correct and none of them says how much trouble the users are in.

This article is about replacing that arrangement with three connected pieces: a measurement that reflects what users experience, a target for that measurement, and alerts that fire when the target is being consumed faster than the team can afford. The pieces are the service level indicator (SLI), the service level objective (SLO), and the error budget, and the alert rule that ties them together is built on the burn rate. The last part of the article turns to the tradeoff the budget exists to settle: how much release risk a service can take on given the reliability it has promised.

The previous article in this series, Instrumenting Services with Metrics, Logs, and Traces, covered how a service emits telemetry: counters and histograms, labels and their cardinality, sampling, trace propagation, and privacy controls. This article assumes you took those away and that your service already emits request counters and latency histograms. It does not repeat how to instrument. It starts from the data and asks what to compute from it and when to wake a person.

All examples are illustrative. The services, traffic levels, and error rates are invented for this article. The Python simulation in the alerting section was written for it and run locally with Python 3.12 and NumPy 2.2; its results are reported as produced. The other calculations are small enough to check by hand. They demonstrate mechanisms and are not measurements of any real system. The Prometheus query and rule snippets were not executed here, because no Prometheus server was available; they are shown as configuration sketches whose semantics come from the documentation cited in the references.

What an objective is for

An SLI (service level indicator) is a carefully defined quantitative measure of some aspect of the level of service provided. An SLO (service level objective) is a target value or range for that indicator. Google’s SRE book states both definitions and adds the service level agreement (SLA), which attaches explicit or implicit consequences to missing an objective (Service Level Objectives). An SLO can exist without an SLA. Internal platforms, pipelines, and APIs benefit from stated objectives even when no customer contract exists.

An error budget is the failure the objective permits. For a ratio indicator with target \( s \), the permitted failing fraction is \( 1 – s \). With \( N \) eligible events in the window, the allowance is \( (1 – s) \times N \) failures. At 99.9 percent, one request in a thousand is allowed to fail. The budget is useful because it converts an abstract reliability promise into a quantity that can be spent, compared, and reasoned about: an incident consumes some of it, a risky release may consume some of it, and a policy can say what happens when it runs out.

Three questions organize everything that follows, and the order matters.

• What counts as the user being failed? This is the SLI definition, and every later number inherits its mistakes.
• How much failure is acceptable over what period? This is the SLO, and the budget follows from it by arithmetic.
• At what speed of consumption does a human need to act, and what should the human do? This is the alerting design, and its test is whether each page leads to an action.

User-visible failures: what to count

The SRE monitoring chapter separates symptoms from causes: what is broken, and why (Monitoring Distributed Systems). The distinction depends on where you stand, since a database being slow is a cause from the service’s point of view and a symptom from the database team’s. An SLI is a symptom measurement taken at a boundary a consumer cares about. It should not be a cause measurement such as CPU utilization or queue depth, even though causes remain essential for diagnosis. The chapter’s four golden signals (latency, traffic, errors, and saturation) are the starting vocabulary for user-facing systems. For SLIs, the first three usually supply the indicator and saturation usually supplies diagnosis and prediction.

“The user was failed” covers more than HTTP 500 responses. The table lists the failure classes that commonly matter for a request-driven service, who can see each one, and where a naive measurement misses it.

Failure class	What the user experiences	Where it is visible	Typical blind spot
Server error	An error page or failed call	Application metrics, load balancer	None if counted, but 5xx caused by the client’s malformed input may not be the service’s failure
Rejected before the application	Connection refused, 429 or 503 from a proxy	Load balancer, client	Application metrics never see the request
Timeout	The client waits, then gives up	Client, load balancer (as a timeout or canceled request)	The server may later complete the work and log a success
Slow success	A correct answer that arrives too late	Latency histograms	An average hides it; a success counter counts it as good
Wrong answer	A 200 response with incorrect content	Only a correctness check or reconciliation	Latency and availability indicators both pass
Stale answer	Correct once, but out of date	Freshness measurement	Request-level success is unaffected

Two consequences follow. First, the good-event rule should usually combine success and timeliness: a good request is one that returned a non-error response within a stated time. The SRE monitoring chapter makes the related point that latency of failed requests and latency of successful requests must be distinguished, and that a slow error is worse than a fast one, so errors should not simply be filtered out of latency tracking. Second, different system types need different indicators. The SRE book groups them: serving systems care about availability, latency, and throughput; storage systems about latency, availability, and durability; data pipelines about throughput and end-to-end latency, which in practice often means freshness; and every system about correctness. A pipeline’s SLI might be the fraction of hours in which a table was no more than two hours behind its source, and a request counter would say nothing about that.

A rule for choosing among candidates, which is the author’s synthesis of the sources and not a published checklist: prefer the indicator that would be the first thing a user complained about, measured as close to the user as is affordable. If you cannot name the user-visible effect of an indicator moving, it is probably a diagnostic metric and belongs on a dashboard, not in an objective.

Measuring an SLI

Specification, implementation, and the ratio form

The SRE workbook separates the SLI specification, the outcome you think matters to users independent of how it is measured, from the SLI implementation, which pairs the specification with a particular measurement approach (Implementing SLOs). “Home page requests complete in under 300 milliseconds” is a specification. “The fraction of requests counted at the load balancer whose latency bucket is at most 0.3 seconds” is one implementation. The same specification can have several implementations with different coverage and engineering cost, and the gap between them is where many disputes about whether the objective was met originate.

The workbook recommends expressing indicators as a ratio of good events to valid events, and the ratio form earns its place because it forces two decisions:

\[ \text{SLI} = \frac{\text{good events}}{\text{valid events}} \]

What makes an event good, and what makes an event valid? Validity decides the denominator. Are health checks counted? Are requests the client canceled? Are malformed requests that were correctly answered with a 400 counted as good, or excluded? Are requests throttled with a 429 because a caller exceeded a quota the service’s failure or the caller’s? Each answer is defensible, and each changes the number. The rule is to decide, write it down, and apply it identically in the measurement, the alert, and the reports. With no valid events in a window, the ratio is undefined, and you need an explicit no-data policy rather than silently reporting success or failure.

Where to measure

The workbook lists the main measurement points with their strengths and weaknesses. The comparison below follows those descriptions.

Measurement point	Sees	Misses	Cost
Application server logs or metrics	Everything that reached the application	Requests that failed before reaching a backend	Often lowest, because the data usually exists
Load balancer or proxy	More of the failures users see, including requests rejected before the application	Failures between the load balancer and the user; client-side behavior	Requires the metrics to be exported and aggregated
Synthetic probes (black-box)	Network-level and end-to-end problems on the paths they exercise	Failures that affect only some users or paths the probe does not use	Moderate; the probes must be maintained
Client-side instrumentation	The experience at the user boundary	Nothing at the boundary, but only for clients that report	Highest; needs code changes, collection, and sampling decisions

No single point is correct. A common progression is to start with the load balancer, which is cheap and covers most failure classes, add a synthetic probe for the paths whose failure would otherwise go unseen, and add client-side measurement where the service’s own view is known to diverge from the user’s. Whichever point you choose, document it, because the objective is only meaningful relative to where it was measured.

A counter-based SLI

For a request-driven service, the implementation is typically two counters: eligible requests and good requests. In Prometheus terms, assuming a request counter named http_requests_total with a code label (names assumed for this sketch, and not executed here), an availability-style ratio over a window might be recorded as:

# Fraction of valid requests that were good over 5 minutes (sketch; metric and label names are assumptions)
sum(rate(http_requests_total{job="checkout", code!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

The expression treats every non-5xx response as good and counts every request as valid. If health checks share the counter they would need to be excluded by a label matcher, and if 4xx responses caused by the service (for example, a validation rule that wrongly rejects valid input) are possible, the matcher is wrong. The sketch makes the point that the SLI is configuration with decisions embedded in it, so it should be reviewed like code. The label design also matters: every additional label multiplies the number of time series, which is why the metric cardinality constraints from the previous article apply to SLI counters too. An SLI should be computed from a small number of well-chosen labels, not from per-user or per-request identifiers.

A latency SLI from histogram buckets

A latency SLI is best phrased as a fraction: the proportion of requests faster than a threshold. That is the same ratio form with “good” meaning “completed in at most the threshold”. Prometheus classic histograms store cumulative counters by bucket upper bound (le), so the numerator is the bucket counter at the threshold and the denominator is the total count. The Prometheus documentation gives this expression and warns that it strictly requires a bucket boundary at the chosen threshold, with no interpolation applied if the boundary does not exist (Histograms and summaries).

sum(rate(http_request_duration_seconds_bucket{job="checkout", le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{job="checkout"}[5m]))

The consequence is a design rule: choose bucket boundaries at the thresholds your SLOs use, before you need them, because bucket boundaries cannot be changed retroactively for data already recorded. Estimation from buckets is a separate matter from the fraction. A percentile estimated with histogram_quantile interpolates linearly inside a bucket, so its error is bounded by bucket width, and the Prometheus documentation shows that coarse buckets can give an estimate far from the true value.

A small calculation, with assumed counts, shows the problem. Suppose 10,000 requests were recorded in a classic histogram with buckets at 0.1 s, 0.5 s, and 1 s, and the cumulative counts are 6,000 at or under 0.1 s, 9,800 at or under 0.5 s, and 10,000 in total. The SLI threshold is 0.3 seconds, which is not a bucket boundary. Linear interpolation inside the 0.1 to 0.5 second bucket assumes the 3,800 requests in it are spread evenly over its 0.4 second width, so it places \( 3{,}800 \times (0.3 – 0.1)/0.4 = 1{,}900 \) of them below 0.3 seconds and estimates \( (6{,}000 + 1{,}900)/10{,}000 = 79\% \) good. The data itself only bounds the true value: anywhere from 6,000 to 9,800 requests, that is 60 to 98 percent, may have been at or under 0.3 seconds. If most of the bucket’s requests sit just above 0.1 seconds, the true fraction is near 98 percent, and a 97 percent objective would appear to be failing badly while being met. If most sit just below 0.5 seconds, the opposite error occurs. An interpolated percentile has the same weakness, since its error is bounded only by the width of the bucket it falls in.

Averaging percentiles fails for a related reason. Take two pods, one serving 1,000 requests with a 99th percentile of 2.0 seconds and one serving 199,000 requests with a 99th percentile of 0.8 seconds. Their simple average is 1.4 seconds, while the pooled 99th percentile over all 200,000 requests is dominated by the large pod and is close to 0.8 seconds, because the small pod contributes only 0.5 percent of the requests. The Prometheus documentation makes the same point about summaries: aggregating precomputed quantiles rarely makes sense, while histogram buckets can be summed across instances and the fraction or quantile computed afterward. The mean also hides the tail: a distribution can have a mean near 0.1 seconds and a 99th percentile near 0.9 seconds, which is why the SRE book recommends reasoning about distributions, and the percentile entry covers the calculation conventions. For SLIs, the design rule is counters and histogram buckets that can be summed, never per-instance percentile gauges.

Low traffic and missing data

A ratio over a small denominator is noisy. If a service receives ten requests per hour, a single failure gives an hourly error ratio of 10 percent, which is 100 times the 0.1 percent a 99.9 percent objective allows. Over a 30-day month that service receives about 7,200 requests, so the whole month’s budget is about 7.2 failed requests, and one failure uses roughly 14 percent of it. Neither the ratio nor a burn-rate rule tuned for high-traffic services is well behaved here. The options are to lengthen the windows until they contain enough events, to aggregate several low-traffic endpoints into one SLI where they share a user promise, to add synthetic traffic that exercises the path, or to decide that the objective is expressed in a different form (for example, a count of failed operations per month). None of these is free, and which fits depends on how much a single failure matters to the user. The choice should be explicit.

Choosing the target and the window

What a target is allowed to be

The SRE book offers guidance on choosing targets. Do not simply adopt the current performance, which locks you into a level that may be accidental and expensive to maintain. Keep the SLO simple and the number of SLOs small enough that each one carries real meaning. Start from what users need and what the business can afford, and accept that perfection is neither achievable nor desirable, because the pursuit of it blocks change. The budget is the explicit form of that acceptance.

Two arithmetic facts help calibrate. Each additional nine reduces the permitted failing fraction tenfold. For a 30-day window, which is 43,200 minutes, the equivalent of full-outage time allowed is computed by multiplying by \( 1 – s \):

Target	Failing fraction	Equivalent full-outage minutes in 30 days
99%	1%	432
99.9%	0.1%	43.2
99.95%	0.05%	21.6
99.99%	0.01%	4.32

A request-based objective uses request counts instead of minutes. Assume, for illustration, a service handling 500 requests per second on average for 28 days. That is 500 × 86,400 × 28 = 1,209,600,000 requests, and a 99.9 percent target permits 1,209,600 failures. The equivalence between minutes and requests holds only when traffic is uniform, and traffic is not. A one-minute total outage at the daily peak costs more requests than one at 3 a.m., so request-based and time-based measures disagree about the same incident, a point also made in the availability entry.

The dependency ceiling

A service’s objective is bounded by what its critical dependencies deliver. If a request needs three dependencies to succeed and they fail independently, the success probability is the product of their individual success probabilities. With assumed availabilities of 99.9, 99.95, and 99.9 percent, the product is 0.999 × 0.9995 × 0.999 ≈ 0.9975, about 99.75 percent, which is below the 99.9 percent target you might want for the service. The calculation is an inference under the independence assumption, and real failures are often correlated, which can make the true figure better or worse in ways the product does not show. The practical use is as a sanity check: before promising 99.9 percent, list the serial dependencies, find out what they promise, and decide whether to soften the dependency (caching, fallbacks, asynchronous handling) or lower the target.

Window choice

A rolling window and a calendar window behave differently at their boundary. A calendar window hands out a fresh budget on the first day of the month regardless of what happened on the last day of the previous month, and the burn-rate arithmetic below works equally well for either window as long as you state which one you use. Rolling windows are common for alerting and policy because they do not reset abruptly. The window also sets the horizon for action: a 28 or 30-day window means a bad week is visible for weeks, which is useful for release policy and slow for a team that wants quick feedback, so alerts use much shorter windows than the objective itself.

Burn rate: the quantity alerts should use

Definition and meaning

The burn rate normalizes an observed error ratio by the ratio the objective allows:

\[ \text{burn rate} = \frac{\text{observed error ratio}}{1 – s} \]

A burn rate of 1 means the service is consuming budget exactly as fast as the objective allows, so that the budget lasts the full window. A burn rate of 10 means ten times as fast. The measure makes alerts comparable across services: a 1 percent error rate means very different things against a 99 percent objective (burn rate 1) and a 99.99 percent objective (burn rate 100). The burn rate glossary entry covers the arithmetic with different traffic assumptions; this section shows how to use it to design alert thresholds.

For a full budget at a constant burn rate \( b \) over a window of length \( T \), the budget lasts \( T / b \). Against a 30-day window, the table converts rates to time, assuming the budget starts full, the error ratio stays constant, and the traffic stays at its average:

Burn rate	Error ratio at a 99.9% target	Time to exhaust a full 30-day budget
1	0.1%	30 days
2	0.2%	15 days
6	0.6%	5 days
14.4	1.44%	about 50 hours (2.08 days)
36	3.6%	about 20 hours
1,000	100%	about 43 minutes

The last row reframes an outage. A total failure against a 99.9 percent monthly objective gives the team about 43 minutes before the month’s allowance is gone, which is far less than the “a day to respond” many teams assume. The column heading’s conditions matter. A request-based budget is a count, so the time depends on traffic, and a budget partially spent has proportionally less time left.

Deriving thresholds from budget fractions

Rather than choose a burn rate and hope, choose a tolerable fraction of the budget to consume over an observation window and derive the threshold. If the window is a fraction \( w \) of the SLO period and you want to alert when a fraction \( f \) of the budget is consumed within it, the threshold at average traffic is \( f / w \). One hour is 1/720 of 30 days, so consuming 2 percent of the budget in one hour corresponds to 0.02 / (1/720) = 14.4. Five percent in six hours gives 0.05 / (6/720) = 6. Ten percent in three days gives 0.10 / (3/30) = 1. These are the parameters the SRE workbook proposes as starting points for a 99.9 percent objective, and it describes them as adjustable to the service (Alerting on SLOs). The derivation is more important than the constants: a different service might choose 1 percent in one hour, and the same formula gives 7.2.

The “average traffic” caveat matters here. A burn rate computed from a ratio is independent of traffic volume, but the fraction of the monthly budget consumed in a window depends on how much of the month’s traffic fell inside it. A burst at peak hour consumes a larger share of the budget than the same ratio at a quiet hour. The thresholds derived from \( f / w \) are exact only under uniform traffic.

Alert design: five rules and what each gets wrong

The workbook evaluates alerting strategies on four properties: precision (the fraction of notifications that correspond to significant events), recall (the fraction of significant events that produce a notification), detection time (how long after the start of the event the alert fires), and reset time (how long the alert keeps firing after the problem is over). It walks through successive designs, each fixing a defect of the previous one.

1. Alert when the error ratio over a short window exceeds the SLO’s allowed ratio. Detection is fast, but precision is low. With a 99.9 percent objective and a 10-minute window, an error ratio of 0.1 percent consumes only a small fraction of the monthly budget, yet the rule would notify on every such window. The workbook shows this can produce up to 144 notifications per day while the objective is still met.
2. Lengthen the window. Precision improves because short blips no longer trigger, but the reset time becomes the window length: with a 36-hour window the alert keeps firing for about 36 hours after the problem is fixed.
3. Add a duration clause (the alert must hold for a period). The Prometheus for clause keeps an alert in the pending state until the condition has held continuously for the duration. If the condition momentarily improves, the timer resets, which hurts recall and detection time for errors that fluctuate around the threshold.
4. Alert on burn rate with one window. Precision and detection improve because the threshold is tied to budget consumption. Recall is limited: a threshold tuned for fast burns misses slower ones. The workbook’s example is that a threshold of 36 never fires for a constant 35, even though 35 would exhaust a full 30-day budget in about 20.5 hours.
5. Several burn-rate thresholds with different windows. Cover fast and slow burns with different severities. The cost is duplicate notifications for one incident, which needs grouping or inhibition.

The recommended design is a multiwindow, multi-burn-rate configuration: each rule requires a burn rate above the threshold in both a long window and a short window. The long window establishes that the consumption is significant; the short window confirms it is still happening, which shortens the time the alert stays firing after recovery. The workbook’s guideline is that the short window is about one-twelfth of the long window.

Severity	Long window	Short window	Burn rate	Budget consumed in the long window
Page	1 hour	5 minutes	14.4	2%
Page	6 hours	30 minutes	6	5%
Ticket	3 days	6 hours	1	10%

The first rule catches fast, severe failures within minutes to an hour. The second catches a slower burn that still threatens the budget within days. The third catches a sustained low-level burn that would exhaust the budget slightly before the window ends, and it files a ticket because the response should not wake anyone.

A simulation to test the rules

The following script tests five rules against five scenarios. The assumptions are explicit: a 99.9 percent objective over 30 days, steady traffic of 1,000 requests per minute (about 16.7 per second), a baseline error probability of 0.03 percent, errors drawn per minute from a binomial distribution (seed 7), and a 40-day timeline with each incident starting on day 10. The five rules are the naive 5-minute ratio rule, a 36-hour window, a single-window burn-rate rule at 14.4 over one hour, the multiwindow page rules (14.4 over 1 hour and 5 minutes, or 6 over 6 hours and 30 minutes), and the multiwindow ticket rule (1 over 3 days and 6 hours). A rule fires in a minute when its condition is true in that minute, and the script measures the delay from the incident start to the first firing minute and the delay from the incident end to the first minute in which the rule stops firing.

Choosing windows and thresholds is done by replaying data through candidate rules, which is what the program does. It was run with Python 3.12 and NumPy 2.2. It generates its own traffic from a fixed seed, so the output below is one draw, and the detection times for S2 and S3 can be compared with the formula given after the tables. The traffic level matters: with a much smaller requests-per-minute value, every window holds fewer events and the naive rule becomes far noisier.

import numpy as np
rng=np.random.default_rng(7)
SLO=0.999; BUDGET=1-SLO
RPM=1000                      # assumed steady traffic, requests per minute
DAYS=40; N=DAYS*1440

def series(base_err, incidents):
    """per-minute (total, errors). incidents: list of (start_min, dur_min, err_ratio)."""
    p=np.full(N, base_err)
    for s,d,e in incidents: p[s:s+d]=e
    tot=np.full(N,RPM)
    err=rng.binomial(tot,p)
    return tot,err

def window_ratio(tot,err,w):
    ct=np.concatenate([[0],np.cumsum(tot)]); ce=np.concatenate([[0],np.cumsum(err)])
    idx=np.arange(1,N+1); lo=np.maximum(idx-w,0)
    return (ce[idx]-ce[lo])/(ct[idx]-ct[lo])

def rules(tot,err):
    r={w:window_ratio(tot,err,w) for w in (5,30,60,360,2160,4320,60*36)}
    out={}
    out['A naive: 5m ratio > 0.1%']= r[5]>BUDGET
    out['B long window: 36h ratio > 0.1%']= r[60*36]>BUDGET
    out['C burn 1h > 14.4 (single window)']= r[60]>14.4*BUDGET
    page1=(r[60]>14.4*BUDGET)&(r[5]>14.4*BUDGET)
    page2=(r[360]>6*BUDGET)&(r[30]>6*BUDGET)
    tick=(r[4320]>1*BUDGET)&(r[360]>1*BUDGET)
    out['D multiwindow page (14.4 or 6)']= page1|page2
    out['E multiwindow ticket (3d & 6h > 1x)']= tick
    return out

def episodes(b):
    d=np.diff(np.concatenate([[0],b.astype(int)]))
    return list(np.where(d==1)[0])

start=10*1440
scen={
 'S1 total outage 10 min':[(start,10,1.0)],
 'S2 2% errors for 3 h (burn 20)':[(start,180,0.02)],
 'S3 0.7% errors for 2 d (burn 7)':[(start,2*1440,0.007)],
 'S4 0.15% errors for 10 d (burn 1.5)':[(start,10*1440,0.0015)],
 'S5 no incident, baseline 0.03% errors':[],
}
for name,inc in scen.items():
    tot,err=series(0.0003,inc)
    res=rules(tot,err)
    print('\n'+name)
    if inc:
        st,d,e=inc[0]; end=st+d
        print('  rule | detect_delay_min | reset_delay_min_after_incident_end | firing_episodes_started_during_incident')
        for k,b in res.items():
            ep=[t for t in episodes(b) if st<=t<end]
            first=(ep[0]-st) if ep else None
            after=np.where(~b[end:])[0]
            reset=int(after[0]) if len(after) else None
            print(f'  {k} | {first} | {reset} | {len(ep)}')
    else:
        print('  rule | firing_episodes_in_40_days | minutes_firing')
        for k,b in res.items():
            print(f'  {k} | {len(episodes(b))} | {int(b.sum())}')

The results below are the script’s output for one seed, rearranged into tables. Delays are in simulated minutes. “Detect” is the delay from the incident start to the first firing; “Reset” is the delay from the incident end until the rule stops firing. A dash means the rule never fired during the incident.

Scenario	Rule	Detect	Reset
S1: total outage, 10 min (burn 1,000)	A. 5-minute ratio	0	4
	B. 36-hour window	1	2,158
	C. 1-hour burn rate 14.4	0	59
	D. Multiwindow page	0	29
	E. Multiwindow ticket	3	359
S2: 2% errors, 3 h (burn about 20)	A. 5-minute ratio	0	4
	B. 36-hour window	75	2,085
	C. 1-hour burn rate 14.4	42	17
	D. Multiwindow page	42	20
	E. Multiwindow ticket	150	347
S3: 0.7% errors, 2 days (burn about 7)	A. 5-minute ratio	1	4
	B. 36-hour window	231	1,928
	C. 1-hour burn rate 14.4	not detected	n/a
	D. Multiwindow page	317	4
	E. Multiwindow ticket	465	323
S4: 0.15% errors, 10 days (burn about 1.5)	A. 5-minute ratio	3	2
	B. 36-hour window	1,255	927
	C. 1-hour burn rate 14.4	not detected	n/a
	D. Multiwindow page	not detected	n/a
	E. Multiwindow ticket	2,477	140

Scenario S5 has no incident and only the 0.03 percent baseline for 40 days. The naive 5-minute rule started 114 separate firing episodes, which is about three per day on average, while the 36-hour rule, the burn-rate rule, and both multiwindow rules never fired. In S4 the naive rule started 1,066 separate episodes during the 10-day incident, about 107 per day, because a 0.15 percent average error ratio sits just above its 0.1 percent threshold and random variation in a 5,000-request window pushes it back and forth across the line.

Reading the results

Several mechanisms are visible, and they match the workbook’s analysis.

Detection time follows from window length and severity. For a constant error ratio, a rule with window \( W \) and threshold ratio \( \theta \) fires when the window’s average, a blend of baseline and incident ratios, crosses \( \theta \). Solving for the time \( t \) after incident onset with incident ratio \( p \) and baseline \( p_0 \) gives \( t \approx W (\theta – p_0) / (p – p_0) \). For S2, the one-hour window, \( \theta = 1.44\% \), \( p = 2\% \), and \( p_0 = 0.03\% \) predict about 43 minutes, and the simulation fired at 42. For S3, the six-hour page rule with \( \theta = 0.6\% \) and \( p = 0.7\% \) predicts about 306 minutes against 317 observed, the difference coming from random variation and the requirement that the 30-minute window also be above threshold. The practical reading: a rule cannot detect faster than its window allows, so fast detection needs the combination of a short window and a high threshold, and a lower threshold is only reachable with a longer window.

A single window has a reset delay equal to its length. Rule B stays firing for about 36 hours after each incident ends, which defeats an alert’s use as a signal of current trouble. Rule C resets within the hour. The multiwindow design cuts this further: in S2, D resets 20 minutes after the incident while its long window would take longer to clear, because the short window drops below the threshold once errors stop. The reset is not instant: the short window still needs to flush the errors out, and the ticket rule’s 6-hour short window resets in about six hours. The workbook’s observation holds: the short window sets how quickly the alert stops, and each row has its own reset time.

A single burn-rate threshold misses slow burns. In S3 and S4, rule C never fired. Both incidents burned the budget at a rate that would have exhausted it in days rather than hours, and a rule tuned for fast burns does not see them. This is the workbook’s 36-versus-35 point at a larger scale. Rule D caught S3 after about five hours and rule E caught S4 after about 41 hours, and S4 is exactly the case the ticket rule exists for: a sustained 1.5 times burn would have consumed the 30-day budget in 20 days, and the correct response is a ticket during working hours, not a page.

Near the threshold, a rule can flap. In S3, rule D started 21 separate firing episodes during the 2-day incident, because the 0.7 percent incident ratio (burn about 7, with baseline) sits close to the 6 times threshold of the 6-hour rule, and random variation moves the windows across the line repeatedly. This is a real design issue: the system pages the same incident repeatedly, which is the pattern behind alert fatigue. The mitigation lies in the alert manager (grouping, repeat intervals, and inhibition so that a firing page suppresses the related ticket) and not in the burn-rate arithmetic. The Prometheus alerting rule supports a keep_firing_for field that keeps an alert firing for a period after its condition stops, which the documentation describes as a way to avoid flapping alerts. Whether to use it, and what to set, is a design choice to test in the same way as the thresholds.

The naive rule is fast and useless. Rule A detects almost instantly in every scenario, but its precision is poor. In S5, it started 114 episodes without any incident, and in S4 it fired about 107 times a day. High recall with low precision is exactly the profile that trains people to ignore pages.

The limits of the experiment deserve stating. One seed and one traffic level were used, so the exact delays and episode counts will vary with both, and the lower the traffic, the noisier every window. The traffic was uniform, while real traffic has daily cycles that change how much budget a window of fixed error ratio consumes. The incidents had constant error ratios, while real failures ramp, oscillate, and affect only part of the traffic. Treat the table as a demonstration of mechanisms, not as a benchmark of the rules. For a real service, replay recorded traffic and incidents through candidate rules before adopting them.

Expressing the rules in Prometheus

In Prometheus the multiwindow rule is an expression that joins two window ratios with and. The workbook’s example uses recording rules for the window ratios and compares them with the threshold times the allowed error ratio. The sketch below follows that structure; the recording rule names are placeholders and the rule has not been executed here.

# Sketch only: recording-rule names and labels are placeholders; not executed in this article.
groups:
- name: checkout-slo-burn
  rules:
  - alert: CheckoutErrorBudgetFastBurn
    expr: |
      (
        checkout:slo_errors_per_request:ratio_rate1h  > (14.4 * 0.001)
        and
        checkout:slo_errors_per_request:ratio_rate5m  > (14.4 * 0.001)
      )
      or
      (
        checkout:slo_errors_per_request:ratio_rate6h  > (6 * 0.001)
        and
        checkout:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
      )
    labels:
      severity: page
    annotations:
      summary: "Checkout is consuming its 30-day error budget faster than the objective allows"
      runbook: "link to the runbook for checkout error-budget burn"

The for clause is deliberately absent. In Prometheus, a rule with for stays in the pending state until the condition has held for the whole duration, and the documentation says that the alert only fires after it remains active throughout it. For a burn-rate rule, the windows already provide the persistence, and adding for delays detection further and introduces the timer-reset behavior described in the third approach above. Whether a particular rule benefits from a for clause is a measurable question in the same simulation.

Two practical points apply. The recording rules should be kept as cheap as possible. Long-range queries process many samples, and precomputing the window ratios at a regular interval trades storage and evaluation cost for query cost. And the ratios should be computed from counters or histogram buckets that can be summed across instances, with consistent handling of counter resets and missing series. A no-data condition should have its own alert, since an absent series makes the ratio expressions return nothing and an empty result does not fire.

Alert action: what a page is for

An alert design that detects the right events can still fail if the page does not lead to a useful action. The SRE monitoring chapter proposes a test for paging rules: does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible? Can the responder do something about it, and does the response require judgment, or could it be automated? A page that fails the test belongs in a different channel, such as a ticket, a dashboard, or an automated remediation.

Applying the test to the burn-rate design gives a mapping from severity to channel and response:

• Page when the burn rate means the budget will be gone within hours or a day or so. The responder’s job is to stop the damage first and diagnose second: roll back the last change, shift traffic, or shed load. The SRE workbook’s incident response guidance makes the same ordering point, mitigating the impact first and finding the root cause afterward. The rollback entry notes that a rollback is only useful if the previous version still works with current data and interfaces, which is a reason to verify the rollback path before it is needed.
• Ticket when the burn rate means the budget will run out in days. The response is planned engineering work during working hours, not an interruption of sleep.
• No alert when the cause is internal and not yet user-visible. CPU saturation, queue depth, and replication lag stay on dashboards and in diagnosis. An exception is a cause that is certain and imminent, such as a disk that will fill in four hours, which the monitoring chapter treats as part of saturation.

What an alert should carry matters as much as when it fires. A useful notification states which SLO is burning, at what rate, what fraction of the budget has been consumed, since when, and where to look first: a link to the dashboard that decomposes the SLI by the dimensions most likely to explain it (region, endpoint, release version, dependency), and a link to a runbook with the first mitigation steps. The Google SRE book’s introduction reports that thinking through and recording best practices ahead of time in a playbook produces roughly a threefold improvement in mean time to repair, a claim about Google’s experience that you should treat as a reason to try it and measure the effect in your own setting. The next two articles in this series return to what those mitigation steps look like for overload and for recovery.

Ownership closes the loop. Every alert needs an owner who can change the alert or the system behind it, and a review process that asks, for each page in the last period, whether it was actionable and whether the action could be automated or the page removed. The fraction of pages that required action is a useful measure, tracked next to the number of interruptions per on-call shift. A rotation that is staffed but overloaded with non-actionable pages is an operational debt (see on-call rotation and toil), and the budget policy below gives teams a legitimate way to spend engineering time on reducing it.

Reliability and release tradeoffs

The budget as a shared currency

The SRE book presents the error budget as a way to align the developers’ interest in shipping change with the operators’ responsibility for stability. Both sides evaluate a proposed release against the same stated target and the same counting rules, instead of arguing from incompatible goals. Remaining budget permits risk under the release policy; it does not waive testing or any other release criterion. When the budget is threatened or exhausted, the agreed response can slow releases, pause them, or direct effort at reliability. Failures that fit inside the budget are still harmful to the users who met them. The budget makes risk explicit and does not make outages desirable.

The workbook’s example error budget policy shows what a written agreement contains: its goals and non-goals, what happens when the service is within its objective, what happens when the four-week budget is exceeded, exceptions, and an escalation path. In the example, once the budget is exceeded, all changes stop except for P0 issues and security fixes until the service is back within the objective, and teams then work on reliability. It lists cases in which the freeze does not apply because the team’s own code did not cause the miss, such as a company-wide network failure, another team’s service, load tests that consumed budget, or errors that were miscategorized and had no user impact. It also sets a single-incident rule: an outage that consumes more than 20 percent of the error budget over four weeks calls for a postmortem with at least one P0 action item (Error Budget Policy). These are one organization’s example choices. The structure is the transferable part: the policy names who writes it, who approves it, when it is reviewed, what it triggers, and how disputes are settled, and the cases where the freeze does not apply are written down before anyone needs them.

A worked example: how release exposure spends the budget

The following arithmetic uses the earlier assumption of 500 requests per second over 28 days, which gives a budget of 1,209,600 failed requests at 99.9 percent. Suppose a bad release fails every request it serves and that rollback completes ten minutes after the release starts serving traffic. All numbers are assumptions.

• Released to all traffic at once, the release fails 500 × 60 × 10 = 300,000 requests, which is 24.8 percent of the budget.
• Released to a 5 percent canary and rolled back at the same ten-minute mark, it fails 15,000 requests, which is 1.24 percent of the budget.

The ratio between the two is the exposure ratio, 20 to 1, and it is the reason progressive delivery and the budget belong together: exposure control is what makes frequent releases affordable. The detection time matters as well. The canary only limits the damage if the failure is detected in ten minutes. The canary deployment entry makes a related point: a traffic percentage is an exposure setting and not a statistical justification, and 5 percent of a small amount of traffic may contain too few relevant requests to show a problem. If the canary receives 50 requests during the observation period, even all of them succeeding says little about a rare failure. The same SLI that the objective uses should drive the canary’s stop condition, with the comparison against the baseline fleet defined before the rollout, so that a canary judged as healthy cannot hide a regression that the production SLO would see.

The arithmetic also shows a second use for the 20 percent single-incident rule. A full-exposure bad release that burns 24.8 percent of a four-week budget in ten minutes would trigger it, and a canary that burns 1.24 percent would not, which gives the policy a measurable incentive to improve rollout safety.

When the budget is not being spent

The mirror case is a service that finishes every period with most of its budget unspent. That can mean the objective is looser than users need, which is only a problem if users are actually harmed, or the objective is tighter than the business wants to pay for, in which case the team may be giving up release speed to buy reliability nobody asked for. It can also mean a quiet period, or safer release machinery doing its job. The way to tell is to check whether changes were deferred because of caution and whether shipping them would actually have used the budget. A persistent surplus is a reason to revisit the target and the release cadence, and it should not be treated automatically as a success.

The cost of another nine

Tightening a target from 99.9 to 99.99 percent shrinks the allowed failing fraction by 10 times, from 43.2 to 4.32 equivalent minutes per 30 days. The incidents that fit in 43 minutes, such as a failed rollout with a ten-minute rollback or a short dependency outage, become budget-exhausting events. Meeting the tighter target probably requires work that has a price: automated rollback, redundancy across failure domains, dependencies with better guarantees, and a response time faster than a human can page, assemble, and act. The claim is an inference from the arithmetic and not a measured cost, but it supports the design rule that the SLO should be set at the level users need, and that each increase should be justified by a user or business need that is worth the engineering cost.

Where SLO programs go wrong

Several failure patterns recur, and each one maps to a section above.

• A green SLO with unhappy users. The SLI measures something users do not experience: a server-side success ratio that misses requests rejected at the edge, a latency average that hides the tail, or a success counter that ignores wrong content. The fix is to compare the SLI with another signal of user experience (support contacts, client-side measurement, a synthetic prober) and to change the definition when they disagree.
• An objective nobody agreed to. A target chosen by one team and never accepted by the product owner has no authority when it conflicts with a deadline. The policy has to be agreed before the budget is empty.
• Aggregates that hide segments. A global ratio can be healthy while one region, tenant, or endpoint is failing badly. Where segments have different promises, compute separate SLIs for them, within the constraint that each additional SLI is another thing to maintain and alert on.
• Alerts without actions. The burn-rate arithmetic is correct and the page still goes to someone who cannot do anything. The response needs a runbook, access, and an owner.
• Silent data gaps. A broken exporter makes the ratio disappear. Alert on missing data, and decide what an empty window means for the budget.
• A budget policy with no enforcement. A freeze that nobody observes teaches teams that the budget is advisory. Whether to enforce is a decision for the people who own the policy, and it is better to write a weaker policy that will be followed than a strict one that will not.
• Counting what the service did not cause. Some failures belong to callers, to dependencies, or to load tests. The policy should say which exclusions apply, record them, and review them, since a broad exclusion list can empty the SLI of meaning.

A specification you can review

A small, reviewable record of each objective makes disputes cheaper. The sketch below is an illustrative format, written for this article and not tied to any tool. It states the decisions that otherwise live in people’s heads.

# Illustrative SLO specification (author-written format; not executed or validated by any tool)
service: checkout
sli:
  name: good-checkout-requests
  good: "HTTP status below 500 and latency at most 0.3 s, measured at the load balancer"
  valid: "all requests to /checkout/*, excluding health checks and requests cancelled by the client"
  no_data: "treat an empty window as unknown and raise a missing-data alert"
slo:
  target: 0.999
  window: "rolling 30 days"
alerts:
  page:
    - { long: 1h,  short: 5m,  burn_rate: 14.4 }
    - { long: 6h,  short: 30m, burn_rate: 6 }
  ticket:
    - { long: 3d,  short: 6h,  burn_rate: 1 }
budget_policy:
  owner: "checkout product and platform leads"
  review: "every quarter"
  when_exhausted: "freeze non-critical changes; reliability work takes priority"
  exclusions: "documented per incident and reviewed in the postmortem"

A reviewer can challenge each line: is the load balancer the right boundary, should client cancellations be excluded, is 0.3 seconds what users notice, who owns the policy, and what counts as an exclusion. A specification that survives review is a better foundation for alerts than a threshold copied from another service.

Limits and next steps

What this article establishes: the definitions and arithmetic of SLIs, SLOs, error budgets, and burn rates; the reasons alerting rules based on error ratio over a single window fail on precision, recall, or reset time; and a simulation, with its assumptions stated, showing how multiwindow burn-rate rules behave. What it does not establish: that these particular windows and thresholds are right for your service, since the workbook itself offers them as starting points, or that a program of objectives improves outcomes in any particular organization. Both are matters for measurement in your own environment.

The conditions in which an objective is hard to meet often come from saturation, where a service is asked to do more than it can. The next article, Capacity, Load Shedding, and Resilience Testing, takes up queues, capacity models, overload controls, and fault injection, and covers how to design the service so that an overload spends a bounded amount of budget.

References

• Google, Site Reliability Engineering, Service Level Objectives
• Google, Site Reliability Engineering, Monitoring Distributed Systems
• Google, Site Reliability Engineering, Introduction
• Google, The Site Reliability Workbook, Implementing SLOs
• Google, The Site Reliability Workbook, Alerting on SLOs
• Google, The Site Reliability Workbook, Error Budget Policy
• Google, The Site Reliability Workbook, Incident Response
• Prometheus documentation, Histograms and summaries
• Prometheus documentation, Alerting rules