Monitoring, Alerting, and Diagnosability

A hypothetical night, though not an unusual one. The pager goes off fourteen times between midnight and six. Eleven are the same nightly job failing a step and retrying successfully; two are nodes that healed themselves before anyone opened a laptop; one is real. The real one is read at 04:12 by someone who has stopped reading carefully, because by then the sound means nothing.

Two weeks later the same service has an outage that takes three hours to diagnose. The team has four dashboards, eleven thousand metric series and a log search that works. None of them can answer the question the incident turned on — which of two upstream dependencies was returning stale data — because nobody had ever exported that, and the window in which it could have been captured has closed.

The noisy night and the slow diagnosis expose related design choices: which conditions deserve immediate attention, and which evidence will remain available during an investigation. Both need planning before an incident. Extra storage alone cannot supply observations that were never captured or make repeated low-value interruptions useful.

The scarce resource is somebody’s attention

Google’s SRE book treats paging as a costly interruption: it diverts an engineer from planned work or interrupts personal time and sleep. A page needs enough urgency to justify that cost.

Frequent low-value pages teach recipients to skim or defer the channel. A consequential alert can then receive the same response, extending the outage. This is the operational mechanism of alert fatigue: noise reduces the attention available to distinguish and act on real problems.

The book expresses the human limit as an on-call expectation:

The SRE book describes an on-call limit: each page should deserve an urgent reaction, yet a person can sustain only a few such reactions in a day before fatigue reduces that response.

Its accompanying principles call for actionable pages that need human judgment and encourage teams to remove recurring mechanical responses through engineering. Treat novelty as a prompt to fix repetitive work, rather than a condition that permits ignoring a familiar urgent failure.

For each candidate rule, identify the urgent action, the active or imminent user impact, and whether another notification already covers it. Review cases where an engineer can safely ignore it and change the condition or routing to avoid those cases. An unusual metric value alone does not establish a need to wake someone.

Automate a rote response when its safety can be established, then remove the page it replaces. Until that response exists and has been exercised, a familiar fault can still need a person: a disk that repeatedly fills and blocks writes remains urgent. Keep its runbook and assign the work needed to remove the recurring intervention. The eleven retrying-job pages in the opening deserve the same review against actual impact and deadlines.

What’s broken, and why, are different questions

The SRE book separates identifying a service symptom from explaining its cause. A user-visible failure tells responders that intervention is needed; evidence about dependencies and internal state helps them decide what to change.

Prometheus’s alerting guidance prioritizes symptoms associated with user harm. Paging at every layer for the same latency increase creates duplicate work when a lower-level delay has not affected the service’s own response objective. Choose a paging boundary and retain lower-level measurements for diagnosis.

The service boundary and the urgency of imminent harm determine how to apply that guidance.

The boundary depends on the service being operated. Slow database reads can be the database team’s service symptom and the frontend team’s diagnostic cause. Define whose outcome the alert protects and coordinate routing when the same incident crosses ownership boundaries.

Some causes justify advance notification because their consequences are sufficiently certain and imminent. A disk projected to fill in four hours may warrant intervention before writes fail. The severity depends on the expected impact and time needed to act; this exception also applies when configuring diagnostic dashboards.

Black-box monitoring tests externally visible behavior, while white-box monitoring uses internal measurements. External probes exercise only selected paths but help ground a page in observed symptoms. Internal evidence can expose imminent resource exhaustion and distinguish causes. Measure both sides of an important dependency: the caller’s elapsed time and the callee’s processing time help separate slow execution from delay elsewhere in the path.

The SRE book’s four golden signals are latency, traffic, errors, and saturation. They provide a starting point for a user-facing service; their definitions determine which failures they reveal.

• Measure successful and failed requests separately. Immediate failures can make overall latency look better while users receive less useful service. Slow failures impose both waiting time and an unsuccessful outcome, so retain their latency as well.
• Include failures that return HTTP 200. The SRE book distinguishes explicit errors, incorrect content despite a successful status, and violations of a service policy such as a one-second response limit. A latency objective can therefore contribute to an error-rate SLI. Incorrect content needs a correctness check; end-to-end tests are one useful way to detect it, while status codes alone cannot establish correctness.
• Watch for saturation before full utilization. Queues and response times can deteriorate before utilization reaches 100%. The SRE book suggests short-window tail latency as an early signal; check it alongside resource measurements because latency also has other causes. Queueing, load shedding, and capacity planning develop the response to this evidence.

The average is the wrong number, and the percentile is harder than it looks

The SRE book illustrates a service averaging 100 ms at 1,000 requests per second while 1% of requests take five seconds. The average conceals the experience of that slow group. When a frontend waits for many backend calls, it has more opportunities to encounter a slow one.

Fan-out increases exposure to the tail. If twenty independent calls each have a 1% probability of a slow response, the chance that at least one is slow is 1 − 0.99²⁰, about 18%. Reaching half the requests would take about seventy such calls under the same assumptions. The SRE book recommends bucketed latency counts as a compact way to retain the distribution; approximately exponential boundaries span a broad range without storing each observation.

Prometheus summaries expose precomputed quantiles that cannot, in general, be combined into a service-wide quantile. Averaging the p99 values of forty replicas does not recover the p99 of their combined requests. Preserve raw observations or an aggregatable distribution representation for that calculation. For classic histograms with compatible boundaries, add corresponding bucket counts across replicas and then estimate the percentile. Summary counts and sums can also be aggregated; the restriction concerns their precomputed quantiles.

Prometheus explains the estimation error in different dimensions. A summary configured for 0.95±0.01 constrains rank: its value may lie between the true 94th and 96th percentiles. In the documentation’s example those are 270 and 330 ms, spanning a 300 ms objective. A classic histogram constrains the value through bucket width and interpolates within the bucket containing the requested rank. In another illustrated distribution, observations cluster around 220 ms; a 200–300 ms bucket produces a 295 ms p95 estimate, while a finer resolution gives 228 ms. The wide bucket can therefore make the service look close to its 300 ms limit even though its observed tail is lower. Counts at the boundaries are measured, but the interpolated quantile is an estimate and does not acquire an exact true rank.

Choose histogram boundaries from the question. For a fraction objective—95% of recorded requests within 300 ms—a classic histogram boundary at 300 ms lets you divide the count at that boundary by the total, without interpolating between buckets. With boundaries only at 290 and 310 ms, the recorded counts can be consistent with either compliance or breach. To estimate the percentile itself, use sufficient resolution where it falls. Changing boundaries later cannot recover detail missing from historical buckets and requires care when combining old and new series. See percentile and response latency for interpreting these quantities.

Three kinds of telemetry, and what each one is good at

Logs, metrics, and traces preserve different views of events and populations. Their freshness, detail, and cost depend on the implementation, but those differences affect which questions can be answered during an incident. The basic distinctions are in telemetry signals; the following examples show how to choose among them.

The SRE workbook describes Google’s metrics system as providing timely aggregates and its logs systems as retaining richer event detail with some delay. In those implementations, metrics commonly supported alerts and dashboards while logs supported diagnosis and detailed reports. These are implementation tradeoffs: the same chapter notes exceptions such as real-time logs and high-cardinality metrics. Either signal can lose, duplicate, sample, or miscount events. Assess collection coverage and processing before deciding which provides the more accurate answer.

Telemetry must arrive soon enough for the intended response. The workbook notes that four or five minutes of staleness can materially delay incident handling. Delayed observations can also make a successful intervention appear ineffective, prompting another change or a mistaken causal explanation. Set freshness and query-latency requirements from the detection and diagnosis tasks they support.

The workbook gives two useful examples. An App Engine team exported HTTP status codes as a metric label so errors could be graphed and alerted on without repeatedly searching logs; the bounded code vocabulary kept series growth manageable. Another team needed entity IDs that could take millions of values. It retained those details in logs and documented a query script for responders instead of creating a metric series for every ID.

Choose a field’s representation from its use and expected number of distinct values. A small bounded dimension can make a metric useful for aggregation and alerting. Per-request or per-customer identifiers can multiply series beyond the available budget, so logs or traces are often a better place for those details. Their ingestion, indexing, retention, and access costs still need a design.

A trace links operations within a request through timed spans, making it possible to inspect where time was spent across services. Logs and traces become easier to join when instrumentation carries trace and span identifiers. Without those identifiers, other recorded request keys or timing evidence may support limited correlation, but exact linkage is harder and can be ambiguous. Metrics summarize populations rather than reconstructing every request. The context and coverage limits are developed in request tracing.

OpenTelemetry describes observability as the ability to investigate a system’s behavior, including unfamiliar problems, using emitted signals. Useful instrumentation supplies the evidence needed to answer an incident’s questions. Buying a backend or collecting all three signal types does not establish that the relevant operation, version, or outcome was recorded.

The questions a team can answer depend on collection, retention, correlation, access, and analysis. Existing logs can become more useful through a new query or join without changing application code. Conversely, historical evidence that was never retained and cannot be recovered elsewhere cannot be created by adding a dashboard later. The workbook recommends using postmortems to identify measurements that would have shortened diagnosis; the same review should check whether needed evidence existed but was difficult to find or connect.

Alert on how fast the budget is burning

After choosing a symptom or an imminent threat, set the threshold against the harm it represents. The SRE workbook uses substantial error-budget consumption to define a significant event. For a reliability objective and an error budget, the reasoning behind setting them comes first; this section assumes they have been defined.

Compare alert rules by precision (the share of alerts corresponding to significant events), recall (the share of significant events detected), detection delay, and reset delay after recovery. The SRE workbook uses these dimensions to explain why fewer alerts alone do not prove that a rule is better.

Simple error thresholds, longer averaging windows, and persistence timers change precision and response delay in different ways, as the following examples show.

A ten-minute error-rate rule that fires just above the SLO can notify on events too small to justify action. In the workbook’s example, 0.1% errors for ten minutes consume only 0.02% of the monthly budget. Repeated such events could produce 144 alerts each day while the service still meets its SLO even if none receives intervention.

A longer averaging window improves precision but can delay reset after recovery. The workbook’s full-outage example triggers shortly after two minutes and remains active for the following 36 hours. The rule detects substantial harm but also retains the old errors long after the immediate condition has ended.

Requiring an hour of continuous breach creates another problem: even a brief return within SLO resets the persistence timer. In the workbook’s example, complete failure for five minutes every ten minutes never triggers that rule while consuming 35% of the budget. The timer therefore misses serious intermittent harm.

Burn rate divides the observed error fraction by the fraction allowed by the SLO. For a 99.9% objective, 0.1% errors means 1× burn. Starting with a full 30-day budget, constant traffic and error fraction would exhaust it in 30 days at 1×, 15 days at 2×, three days at 10×, and about 43 minutes at 1,000×. These times depend on the assumed request pace: the same error fraction consumes more requests during peak traffic, and the ratio does not say how much budget remains. Estimate actual time to exhaustion from remaining allowance and expected traffic. Burn rate develops the arithmetic and limitations.

The Google accounts describe practice at the time of writing; the recommended burn-rate parameters are starting points the workbook says depend on the service.

As starting points, the SRE workbook suggests paging for consumption equivalent to 2% of a monthly budget in one hour or 5% in six hours, and ticketing for 10% in three days. Under its traffic assumptions, those choices produce the following burn thresholds for a 99.9% objective. Tune them against the service’s consequences and response capability.

Severity	Long window	Short window	Burn rate	Budget consumed
Page	1 hour	5 minutes	14.4	2%
Page	6 hours	30 minutes	6	5%
Ticket	3 days	6 hours	1	10%

Requiring both windows to exceed the threshold reduces the time an alert remains active after recovery. The workbook suggests a short window one-twelfth as long as the long window. Reset time still depends on the short window, preceding errors, and traffic: the table has 5-minute, 30-minute, and 6-hour short windows, so it does not promise that every alert clears within minutes. With steady traffic, a 6-hour window filled by a 5× burn takes 4.8 hours of zero errors to fall to 1×. The ticket row addresses a sustained lower burn; its 10% budget figure over three days assumes the average request pace used for the monthly budget.

Use the two paging branches joined by or; a rule containing only the 14.4× branch misses a sustained 7× burn. At the month’s average request pace, 7× for six hours uses about 5.8% of the budget. The following is a rules: list excerpt inside a Prometheus rule group. It assumes recording rules named below expose failed/eligible request ratios, calculated from the same event definition and matching service/SLO label sets over each window. 0.001 is the allowance for 99.9%. Missing observations and zero-request windows need explicit handling rather than being treated as success. Alert routing uses the additional severity label, as described in the Prometheus alerting-rules reference.

- alert: ErrorBudgetBurn
  # fast:  14.4x over 1h AND still burning over the last 5m  -> 2% of a 30-day budget
  # slower: 6x over 6h  AND still burning over the last 30m  -> 5% of the budget
  expr: |
    (
      slo_errors_ratio_rate1h > (14.4 * 0.001)
        and
      slo_errors_ratio_rate5m > (14.4 * 0.001)
    )
      or
    (
      slo_errors_ratio_rate6h > (6 * 0.001)
        and
      slo_errors_ratio_rate30m > (6 * 0.001)
    )
  labels:
    severity: page

- alert: ErrorBudgetBurnSlow
  # 1x burn sustained over 3 days -> 10% of the budget spent, next working day
  expr: |
    slo_errors_ratio_rate3d > 0.001
      and
    slo_errors_ratio_rate6h > 0.001
  labels:
    severity: ticket

Even with both branches there is a floor: a sustained burn under 1× never makes either rule true. That is the rules’ behavior, not a statement about the month, because every condition above reads a moving window rather than the budget that is left. A service that has already spent almost all of its allowance can stay under every threshold and still miss the objective — 990 of 1,000 permitted failures gone, then a 0.05% error rate (half a burn) across the last 100,000 requests of the month adds fifty more. Whether the month complies is a question for the remaining budget and the traffic still expected; these rules answer only whether the current burn deserves someone’s attention now.

The detection delay is not the length of the long window either. Three days is the period the average is taken over, not a wait before firing. With a clean recent history and steady traffic, a 5× burn pushes the 72-hour average past 1× after about 72/5 ≈ 14 hours — and the six-hour window is already far above the threshold by then, so the ticket fires at that point rather than on day three. The same arithmetic gives about two and a half days for a 1.2× burn and nearly the whole window for a burn just above 1×, and because the comparison is a strict inequality a burn sitting exactly at 1× never fires. What sets the delay is the errors already inside the window, the traffic, and the new rate. Configurations that want the middle of the range sooner add an intermediate row, typically 3× over twenty-four hours with a two-hour short window, at ticket severity.

Overlapping burn conditions can produce several notifications for one incident. The workbook’s example spends 10% of the budget in five minutes, exceeding both slower thresholds as well. Coordinate page and ticket routing or inhibition, and test that suppression ends when its parent condition clears. Similar coordination can replace many per-node alerts with one service-level alert when they represent the same global failure; retain enough detail to locate affected nodes.

The tradeoff is configuration complexity. Record why each threshold and window was chosen, how alerts are grouped, and which SLO each rule measures. Reuse a tested pattern while allowing service-specific choices based on user impact and the team’s response capacity.

When there is not enough traffic to tell

At ten requests per hour, one failure gives a 10% hourly error rate. Against a 99.9% SLO, that is 10% / 0.1% = 100× burn. At a steady ten requests per hour over 30 days, 7,200 requests permit 7.2 failures, so one failure consumes about 13.9% of that allowance. This budget share explains why the rule is sensitive to an individual failure.

A budget permitting only about seven failures per month makes each request consequential to the ratio. Whether that target is justified depends on the value of a failed operation, retries or compensation available to users, and the cost of intervention. Small denominators increase variability; they do not by themselves make the SLO indefensible or justify lowering it.

The workbook offers ways to increase the evidence, reduce user harm, or revise the accepted service expectation. Each changes a different part of the problem.

• Generate artificial traffic. Synthetic requests give the ratio more observations and can reuse existing alerting logic. They may miss real-user paths, however. Combining successful probes with failing real requests can dilute the user failure signal; retain a way to distinguish their populations.
• Combine related services for monitoring. Aggregating low-traffic services yields more observations but can hide the complete failure of a small member. The workbook suggests related failure domains, such as a shared database, as a useful grouping condition. Keep the lost per-service sensitivity explicit.
• Reduce the impact of an individual failure. Client retries with exponential backoff and jitter, or a fallback that captures work for later completion, may reduce user harm. This changes product behavior and needs its own correctness, delay, and capacity checks. If the resulting experience supports a less strict objective, it can provide more error allowance and response time.
• Reconsider the objective with its owners. If a few errors can safely wait, review whether the current paging policy and SLO represent user needs. Lowering the SLO is a product decision that changes the accepted service expectation; monitoring noise alone does not justify it.

Choose among these responses by examining the impact of individual failures. A high-value transaction may warrant investigation after one failure, even though an aggregate burn-rate rule is not the best trigger. A transaction-specific failure alert and reconciliation path can provide that response. The fact that the original request has ended does not remove the need to investigate or repair its effects.

Data and model services fail quietly

Everything above assumes a request-serving system whose failures announce themselves. Pipelines, warehouses and model-serving systems break differently, and the difference is not a detail — their characteristic failure returns a valid-looking answer. Nothing errors, nothing slows down, and the four golden signals stay flat while the output quietly stops being true.

Google’s Rules of Machine Learning describes a joined table that stopped updating while the model continued returning plausible predictions. Quality could deteriorate gradually instead of causing an execution error. In the documented Google Play example, refreshing a table that had been stale for six months increased install rate by 2%. That is a reported case, not an expected gain from every freshness repair.

The same guidance describes a feature whose populated share falls from 90% to 60% after an implementation change. If missing values remain valid inputs, execution can continue without an exception. Track population rates and inspect representative records to detect the loss of coverage.

Freshness, completeness, and distributions reveal failures in the data itself; comparing training and serving also tests whether the model receives the representation it was built for.

• Freshness against an explicit requirement. Google’s ML guidelines ask teams to assess how prediction quality changes as a model ages, from a day through a week or a quarter. That evidence helps prioritize freshness monitoring. For input tables, connect their age to the downstream deadline or model consequence; data freshness explains the measurement boundary.
• Volume and completeness per run. Row counts, null rates and the population rate of each important field, compared with the same run last week rather than with a fixed threshold. This is where the 90%-to-60% coverage change becomes visible, and it is the core of data observability and of data quality checks that run in production rather than in review.
• Distribution, for inputs as well as outputs. A moved input distribution is a reason to look, not a finding on its own: a model that captured the relationship correctly can keep doing so while the mix of inputs changes, and a shifted distribution is just as often a broken upstream transform as a change in the world. What decides the retraining question is what you find next — whether the shift traces to a pipeline fault, whether measured quality against real outcomes has fallen, and which population carries the loss. Data drift and that decision are worked through in monitoring drift and retraining.
• Training-serving skew. Google’s ML guidelines identify both processing discrepancies and changes in data as possible contributors to different training and serving performance. Compare both transforms on the same raw records before release, including missing values, units, time zones, and category vocabularies. Production monitoring is still needed because future population changes cannot be fully established by a pre-release test. Logging the features actually used in serving and reusing suitable samples for training can reduce reconstruction discrepancies, subject to retention, privacy, and label-alignment requirements. It does not remove every source of skew. See training-serving skew.

Batch alerting should follow delivery deadlines and expected runs rather than copying a five-minute request-error rule. Prometheus suggests checking how recently a job succeeded and whether the gap threatens users. Its example allows two full runs: a four-hour schedule with a one-hour runtime gives a suggested ten-hour threshold. That number is suitable only when consumers can tolerate the delay. If one missed run is intolerable, consider a more frequent schedule or another recovery design; increasing frequency still needs enough capacity and safe retry behavior.

That threshold is also why the eleven pages in the opening were wrong: a step that fails and then succeeds with the deadline still comfortably ahead has not reached a user, and what deserves the page is the pipeline’s output being late. Two conditions have to hold before a green run after a retry counts as no impact. The retry must not have left a duplicate effect outside the pipeline — a second file delivered, a notification sent twice, a downstream increment applied again — and the eventual success must land before the consumers’ deadline, because a run that finishes after the report was due is late output wearing a green tick. That makes the batch measures their own set: whether every expected run and partition arrived, how old the newest record is at each hop, whether the output was published rather than merely written, and whether it was published in time. For the model-side measures — feature drift, prediction quality, serving latency — see monitoring pipelines.

Google’s ML guidelines distinguish problems caught before model export, which can use an owned email workflow, from problems in a user-facing model that may require a page. Check candidate quality before release and block promotion when it fails the agreed criteria. Select the channel by urgency and ownership rather than treating every model problem as an overnight interruption.

What observability actually costs

Telemetry costs extend beyond the vendor invoice. Cardinality, sampling, resolution, and retention determine which evidence remains available, while instrumentation and routing also require engineering time.

Metric cardinality multiplies storage and query work because each distinct label set creates a series. Prometheus offers conservative instrumentation guidance: aim for fewer than ten combinations for most metrics and investigate alternatives when a metric can exceed a hundred. These are design prompts, not universal capacity limits. Assess combinations across instances, label churn, retention, scrape frequency, queries, and the backend’s resource budget before retaining or removing a dimension.

Prometheus illustrates this with roughly 100,000 filesystem series across 10,000 nodes, then warns that adding 10,000 users can push the count into tens of millions. That example demonstrates multiplication and opportunity cost; the documentation’s implementation warning is not a timeless benchmark for every deployment. Start with dimensions needed for concrete questions and evaluate additions against the same budget. Metric cardinality develops the arithmetic and diagnostic choices.

Trace sampling can reduce cost, but the required rate depends on the questions and request populations. OpenTelemetry notes that low rates such as 1% can represent common behavior in sufficiently large systems. That does not guarantee capture of rare incidents: under independent 1% sampling, all ten occurrences of a rare event are missed with probability 0.99¹⁰, about 90.4%. Evaluate coverage by relevant class and keep appropriate policies for important errors; filtering a class out entirely answers a different question from sampling it.

Head sampling decides early, often using a trace ID to make a consistent probability decision. Whole-trace consistency requires compatible decisions and context propagation across services, plus successful instrumentation and delivery; sampling cannot repair missing spans. Head sampling also cannot use an error that occurs later. Tail sampling can preferentially retain errors or slow traces using the spans it receives and still holds. It cannot recover traces dropped by an earlier head sampler, and late spans, decision windows, or capacity limits can leave evidence incomplete. OpenTelemetry describes the state and operating cost of tail samplers and recommends monitoring their capacity. Compare those costs with retaining more telemetry, rather than assuming sampling is always cheaper overall.

Measurement resolution and retention should follow the behavior being investigated. A one-minute CPU average can hide a short hotspot, while storing every second indefinitely may be costly. The SRE book describes collecting fine observations into buckets and storing minute aggregates, retaining evidence of brief hotspots without every raw observation. Probe frequency also needs an explicit detection objective: its example of one or two probes per minute for 99.9% is a starting judgment in that setting, not a guarantee of detecting every outage that matters to another service.

Monitoring also consumes engineering time. The SRE book reports one or two monitoring specialists in a Google team of 10–12; that is an organizational example rather than a staffing ratio to apply at every scale. Budget for maintaining instrumentation, queries, rules, storage, and routing under the local operating model.

Review signals and configuration that nobody uses, including rarely exercised rules, before paying indefinitely to retain them. The book treats them as removal candidates, not automatic deletions: infrequent incidents may still require retained diagnostic evidence. Cardinality reduction, aggregation, sampling, and retention changes are other cost controls. Each change should preserve the questions and response obligations the team still needs to support.

The monitoring system is a service you operate

Operating the monitoring service requires controlled configuration changes, tests that notifications reach responders, and checks for failures in the monitoring path itself.

Keep monitoring changes reviewable, reproducible, and reversible. The workbook favors intent-based configuration because it can support version control and repeatable changes. A web UI can also provide audit history, approvals, and rollback; the interface alone does not determine whether a change is recorded. Verify that the chosen workflow records who changed what and allows a known configuration to be restored.

Test the chain from instrumentation through rule evaluation to notification delivery. A rule may remain quiet for months before its condition occurs. Check that exported values change as intended, the rule fires and resolves for representative inputs, and its notification reaches the intended destination. Routing tests expose failures that expression tests cannot.

Monitor the monitoring path itself. Missing telemetry or a broken notification route can look like quiet service unless independent health checks or absence detection distinguish them. Test that distinction, including which channel raises the warning if the primary path fails.

Record deployments and configuration changes so responders can compare them with the incident timeline. The workbook recommends exposing binary versions, command-line flags, and dynamic configuration versions. For dependencies, collect request and response sizes, latency, and status where useful; a shared client library can make that coverage more consistent. These observations help investigate partial failure without assuming that temporal correlation identifies its cause.

Where to start, and what not to build

Start with one important service and expand after testing its measurements and response paths. The effort and order depend on existing instrumentation, ownership, and deployment constraints, rather than a fixed two-week delivery promise.

1. Define a user-visible ratio with an explicit good-event rule and population, and align its instrumentation with relevant failure logs. The SRE workbook reports a team whose SLO-linked alerts became more actionable and produced fewer false positives. Use that case as a reason to evaluate the design against your own incidents, not as a guarantee of the same outcome.
2. Bucket your latency around the objective, not around round numbers, and keep the boundaries stable afterwards.
3. Write the burn-rate rules the table calls for — a page covering both the fast burn and the slower sustained one, and a ticket — then delete the threshold alerts they replace. Before deleting, check that each old alert’s condition falls inside what the new rules detect; the ones that do not are telling you which row your configuration is missing. Deleting is the step that gets skipped, and skipping it means you have added noise rather than replaced it.
4. Add diagnostic evidence that answers “why” when the page fires: what changed, dependency responses, saturation of each resource, and traffic broken down by status. Use these primarily for diagnosis; a sufficiently certain and imminent capacity failure can also justify a targeted alert with a defined response.
5. For every pipeline or model, add freshness, volume and coverage checks against a stated requirement, since none of the above will notice a stale join.
6. After an incident, identify which evidence would have shortened diagnosis and why it was unavailable. Add an appropriate metric, log field, span attribute, or version reference where needed, or improve access and correlation when the evidence already exists. Verify that the change answers the question before treating it as an improvement.

The SRE book also describes choices Google avoided or limited. Their failure modes are useful design questions; they are not universal product prohibitions.

• The SRE book favors understandable rules over opaque threshold learning and automatic causal claims, while allowing simple detection of abrupt traffic changes. Evaluate any automated detector against known incidents, ordinary variation, and the explanation responders need; statistical association alone does not establish a cause.
• Complex dependency-based suppression can encode an architecture that soon changes. Google’s account favors limited uses such as suppressing deliberately drained capacity. Where dependencies do drive suppression, test both the intended reduction in duplicate notifications and the risk of hiding a distinct failure.
• Notifications without ownership or follow-up can accumulate unread, including email aliases. Choose a ticket, email, dashboard review, or page according to urgency and an agreed response process. A pre-export model warning can use email when someone owns and tracks it; the problem is unattended notification, not the transport itself.
• Avoid making unattended production detection depend solely on a person continuously watching a screen. Automated detection and routing free people to investigate; dashboards remain useful during that investigation.

The SRE book’s Bigtable case shows how alert design can obstruct improvement. Its objective used a synthetic client’s mean latency while slow-tail requests dominated the experience. Alerts on both approach and breach generated enough noise that engineers spent substantial effort finding actionable cases and missed some user-impacting problems.

The team worked on performance while temporarily moving the objective to the 75th percentile and disabling an unmanageable email stream. The account reports that the reduced interruption load freed engineers to improve the service. This supports deliberate alert reduction with explicit target decisions and continuing observation; it does not establish that less telemetry or fewer alerts always increases reliability.

In the opening scenario, the retries deserve pages only if their effects or completion deadlines require urgent action; verify both timely publication and the absence of duplicate external effects. A self-healed node need not produce a page when user outcomes and remaining capacity are acceptable. Preserving attention for the real incident also requires retaining the dependency evidence needed to diagnose it. Review both notification decisions and evidence gaps after the event.

Compare page load with the team’s actual response capacity, then inspect incidents that were slow to diagnose. Was the necessary evidence missing, discarded, inaccessible, or difficult to correlate? The answers guide different changes, and the next incident or a targeted exercise can show whether those changes helped.

References

• “Monitoring Distributed Systems,” Google SRE Book
• Google SRE Workbook — Monitoring
• Google SRE Workbook — Alerting on SLOs
• OpenTelemetry — Observability primer
• OpenTelemetry — Sampling
• Prometheus — Alerting
• Prometheus — Instrumentation
• Prometheus — Histograms and summaries
• “Rules of Machine Learning: Best Practices for ML Engineering,” Google for Developers
• Prometheus — Alerting rules