Alert Fatigue

Alert fatigue develops when frequent notifications make recipients less likely to distinguish and act on an urgent one. Google’s SRE account describes engineers skimming or ignoring pages and missing real incidents among the noise. Review the conditions and routing that create that load rather than attributing the failure only to individual attentiveness.

People have limited capacity for urgent interruption. The useful limit depends on staffing, shift length, incident complexity, and recovery time; it is not a fixed count shared by every team. Track actual interruptions and their consequences alongside the fraction of pages that required action.

It is expensive before anybody is tired

A page interrupts planned work during office hours and personal time or sleep outside them. Its cost includes restoring concentration and the engineering work displaced by the response.

Recurring pages can also consume the time needed to remove their cause. The SRE book describes this tradeoff between present operational work and improvements that make later operation easier. Assign remediation capacity while continuing to meet current response obligations.

The SRE workbook illustrates a poorly discriminating rule: with a 99.9% 30-day SLO and steady traffic, an error rate of 0.1% over ten minutes uses only about 0.02% of the monthly allowance. Notifying on every such window can produce up to 144 notifications per day while the overall SLO is still met. The example shows why the rule should reflect consequential impact rather than every small excursion; actual notification counts also depend on grouping and repeat settings.

Where the noise comes from

Several design choices create avoidable notification load. Each needs a different remedy.

• Low-precision thresholds. A condition can be common without requiring immediate action. Relate the trigger to user impact and budget consumption, as in burn rate. Threshold tuning, window changes, or a different signal may improve discrimination; evaluate both missed incidents and unnecessary pages.
• Duplicated symptom and cause alerts. One failure can generate notifications at several layers. Prefer an actionable service symptom for paging and retain internal evidence for diagnosis. A sufficiently certain and imminent cause, such as capacity about to run out, can still justify advance notification.
• Per-instance alerts during fleet failures. Google’s Gmail example involved thousands of tasks, making individual descheduling alerts unmanageable. Aggregate shared incidents and preserve instance detail for investigation. Test suppression and its end condition so it does not hide a later independent problem.
• Uncoordinated recipients. Several teams may need to respond to one incident, but redundant pages without distinct responsibilities add interruptions. Make ownership and escalation explicit.

A low-severity email stream still needs an owner and a review process. The SRE book describes unread email noise and proposes dashboards for ongoing subcritical issues. An owned email or ticket workflow can also work; moving a page to an unattended alias does not establish a response.

Deciding what to delete

Review whether each page is urgent, actionable, and dependent on a person. Use recurring events to identify engineering work that can remove the intervention.

1. Urgency. If the problem can wait within its impact and recovery constraints, route it to a tracked work item rather than an immediate page.
2. Action. State what the responder can investigate, mitigate, or escalate. Information without an urgent response may belong in a dashboard or scheduled review.
3. Human judgment. A safe mechanical response is a candidate for automation. Keep the page until the automatic response exists and has been exercised.
4. Recurrence. A known fault blocking users still needs intervention. Repetition identifies a cause to fix; it does not make the current impact acceptable.

Inspect cases that recipients already ignore and determine why. They may reveal a redundant notification, an incorrect threshold, or an unresolved incident that still needs attention. Remove the rule only after confirming which of those cases applies.

A runbook can make a recurring response safer while automation is being built. Compare the ongoing response cost with the implementation, validation, and maintenance cost of automation. Before retiring the page, confirm that user impact is covered, the response no longer depends on a person, and failure of the automated response itself will be noticed.

Review rarely used configuration and signals with no identified consumer. Infrequent diagnostic or incident needs can still justify retention, so treat lack of a dashboard or recent firing as a question about purpose rather than automatic permission to delete.

The evidence that cutting alerts improves reliability

The SRE book’s Bigtable case illustrates how deliberate noise reduction can free time for service improvements, while also showing why the decision needs an explicit objective.

Bigtable’s objective used a synthetic client’s mean latency despite a slow tail in the request distribution. Alerts on approach and breach produced many interruptions; engineers spent substantial time triaging them and missed some problems that affected users.

Alongside performance work, the team temporarily moved its latency objective to the 75th percentile and disabled an unmanageable email stream. The account reports that engineers could then spend more time improving the service.

The case supports adjusting targets and notifications deliberately while preserving awareness of user impact. It does not show that reducing alerts always improves reliability. Quietly muting a channel without arranging coverage can leave an incident undetected.

The SRE account describes quarterly management reviews of incidents per on-call shift, with related pages grouped into incidents. Use both grouped incidents and the interruptions people actually received to assess load. Review trends with staffing and incident severity rather than treating a single daily count as a universal human limit.

Assign severity according to impact and required response time. A sustained low error rate may warrant a ticket while a broad outage needs immediate action. Prefer signals that distinguish actionable states from ordinary workload variation, and test the threshold against both.

How to choose the symptoms worth paging on, and what to keep for diagnosis instead, is worked through in Only a Few Times a Day.

References

• “Monitoring Distributed Systems,” Google SRE Book
• Google SRE Workbook — Monitoring
• Google SRE Workbook — Alerting on SLOs