Incident Management and Problem Management

A hypothetical shift, assembled from parts that are not. The pager fires at two in the afternoon: one region has stopped serving. The on-call engineer starts reading logs. A second region goes. She reverts the last release; it does not help, so she calls the developer who wrote most of the service, for whom it is half past three in the morning. Two colleagues start “just looking” from their own terminals. An executive rings her manager to ask why he was not told. Somebody several hops away has an idea about CPU affinity, pushes it to production to test it, and the remaining servers restart into the change and die.

Everyone in that scene was doing their job as they understood it. The SRE book’s longer version shows how an overwhelming technical task prevents its owner from tracking the bigger picture, poor communication leaves colleagues unaware of each other’s actions, and well-intended but uncoordinated production changes can worsen the outage. These response failures need coordination as well as a technical repair.

The immediate objective is to reduce user harm without waiting for a complete causal explanation. Choosing a safe mitigation still requires diagnosis: responders need to understand the affected paths, test a working hypothesis, and check that a proposed change will not make the situation worse. Restoring service and reducing the risk of recurrence have different completion criteria, even when some of the work happens in parallel.

Mitigate without waiting for a complete explanation

In Effective Troubleshooting, the SRE book places mitigation ahead of a complete root-cause investigation during a major outage. Its instruction addresses a dangerous delay: pursuing an explanation while an available mitigation could reduce harm.

During a major outage, the SRE book recommends improving the system’s immediate operating state before pursuing a complete root-cause explanation. That protects users while the deeper investigation continues.

Available mitigations may redirect traffic to healthy clusters, discard traffic to contain a cascading failure, or disable expensive subsystems. Choose a safe action that limits current harm; an explanation delivered after the whole service fails cannot help the users still depending on it.

The aviation analogy in the SRE book is to keep control of the aircraft and protect its occupants before fully troubleshooting the emergency. In software, that can mean stopping operation when continued execution risks irreversible data corruption. Restoring activity is not always the safest immediate objective.

Preserve evidence while choosing and applying the mitigation. The SRE book explicitly includes logs and other diagnostic evidence in rapid triage. Before a restart clears volatile state, capture what can be retained without an unacceptable delay in protecting users. If continuing the failure would cause greater harm, mitigate first and record what evidence could not be saved. Stopping the damage, restoring service, and preserving evidence are concurrent responsibilities whose order depends on the incident, not a fixed sequence that always puts evidence last.

This priority can be unfamiliar to engineers accustomed to understanding a defect before changing code. During an outage, time-box the investigation around the next safe action, and delegate deeper analysis when staffing permits. A useful hypothesis can guide mitigation without yet explaining every contributing cause.

Two jobs, two clocks

An incident produces related work with different completion criteria: restoring an acceptable service state and reducing the chance or impact of recurrence. A repair can contribute to both, but completing one does not establish that the other is complete.

	Ending the incident	Removing the cause
The question	How do we make this stop affecting users?	Why did it happen, and what prevents the next one?
Who sets the deadline	User impact and the urgency of containment	An assigned owner and risk-based deadlines, considering recurrence, evidence retention and temporary controls
Acceptable answer	A revert, a traffic shift, a disabled feature, a manual workaround. Ugly is fine	A change to the system or the process, with an owner and a priority
Done when	Agreed urgent recovery criteria are met, the current state is communicated, and residual impacts and follow-up work have named owners; a complete causal explanation can still be pending	The class of failure is less likely or less severe, and you can point at what changed

A mitigation can be shipped before the full causal investigation or postmortem is complete. That permits recovery without waiting for the document; it does not establish a standard two-week investigation allowance. Set the investigation and review deadlines from residual risk, the lifetime of temporary controls, and how long useful evidence will remain available.

Treating recovery and prevention as the same obligation can delay an available mitigation or leave the cause in place after recovery.

• Waiting for complete understanding. Responders defer an available safe mitigation while pursuing a full causal explanation. Keep diagnosis focused on decisions that reduce harm, and continue deeper investigation alongside or after recovery.
• Mitigating and moving on. The revert works, everyone goes back to what they were doing, and the cause stays in the system. This one is invisible until it happens again, which is why it needs an organizational mechanism rather than good intentions — the subject of the second half of this article.

Stand down the urgent response after checking that the agreed recovery criteria are met and communicating the current state. Name an owner for remaining damage, temporary changes and residual risk, with a review or expiry time for each. A restored request path does not itself prove that corrupted data has been repaired or that the same failure cannot recur; those obligations need explicit follow-up. Creating a follow-up ticket does not justify standing down while the urgent recovery criteria remain unmet.

Somebody has to not be fixing it

Google adapts the Incident Command System to software operations. FEMA traces the development of ICS through the interagency FIRESCOPE work of the 1970s. The useful inheritance is a scalable structure for assigning responsibilities and coordinating responders; software teams adapt that structure to their systems and organization.

The SRE workbook groups incident management around coordination, communication and control. These give responders concrete questions to ask: are tasks assigned, are the affected parties informed, and is someone maintaining the overall state? They are a diagnostic framework, not an exhaustive list of everything that can go wrong.

Assign command, technical operations, communications and planning according to the work the response needs; the responsibilities matter more than the titles.

• Incident commander. Maintains the high-level state, sets priorities and delegates responsibilities. The commander retains responsibilities that have not yet been delegated, so one responder can initially hold several roles.
• Operations lead. Coordinates technical diagnosis and operational changes with the commander. Keeping production changes within the operations team prevents uncoordinated interventions.
• Communications lead. Provides periodic updates to responders and stakeholders and handles inquiries. Assign the responsibility explicitly, separating it from operations when the workload warrants another responder.
• Planning. Arranges additional staffing, handoffs and practical support, records follow-up work, and tracks emergency deviations that will need review or reversal after recovery.

Explicit decision boundaries give each responder room to act without anticipating uncoordinated decisions by colleagues. Trust people to exercise authority within their assigned role. In a small incident, one person may initially hold command and all undelegated responsibilities; make the active role and any handoff visible. When reading a stack trace prevents the commander from maintaining the overall state, delegate operations as soon as another responder is available. Keep the production-change path under one named operations responsibility even while the same person also commands, so an operator can rely on colleagues coordinating rather than pushing a competing change.

Support those roles with a recognized command post where stakeholders can find the commander, a live incident document that the commander maintains, and an explicit handoff protocol. Keep incident-management tools independent of the failing service so that responders can still coordinate when it is unavailable.

Transfer command explicitly: the outgoing commander names the incoming commander, obtains a firm acknowledgment before leaving the call, and informs the other responders. Everyone should be able to identify the current decision owner throughout the handoff.

The reason for the theater is that the alternative failure is silent. Two people each believing the other is in charge produces exactly the same observable behavior as nobody being in charge, and it is discovered at the worst moment. See incident commander for what the role does and does not include.

Declare early, because you cannot retrofit coordination

Declare early enough to establish roles and tools before coordination becomes the bottleneck. If the fix proves simple, the response can be closed quickly; assembling an incident framework hours into a growing outage leaves responders without that support when they need it.

Write declaration criteria before responders face stress. One team in the SRE book declares an incident if any of these conditions holds: another team is needed to fix the problem, customers can see the outage, or concentrated analysis has not resolved it after an hour.

These criteria catch different coordination needs. Involving another team creates a coordination cost; an hour of unsuccessful analysis catches a slow-burning problem even when it looks undramatic. Customer visibility covers user impact. Declaring only for visible outages leaves the cross-team and stalled-investigation cases without the same response structure.

In the workbook’s Google Home case, a bug fetched speaker-recognition files at fifty times the expected rate and exhausted a quota. As rollout expanded, users lost half their requests over a weekend. The retrospective judged that earlier escalation would have provided the incident tools sooner and enabled a faster, more organized response, even though the incident ended without a major calamity.

Declaring an incident consumes attention and coordination time; the cost depends on how many people are mobilized and what response level is chosen. Use an early, proportionate declaration when coordination is needed, and scale the response up or down as evidence arrives. A low threshold for naming an owner and opening a shared record need not mean calling everyone for every problem.

Silence reads as inaction

Acknowledge the incident and the work under way so stakeholders know the problem is being addressed. Once it is mitigated or resolved, explicitly call off the response; otherwise people can reasonably assume it continues. The SRE workbook treats both messages as operational responsibilities.

Both halves cost real work. The first produces the executive phoning your manager in the opening scene — not because they are unreasonable, but because in the absence of information the only available action is to ask someone. The second produces a response that never formally ends, with people still watching dashboards for a problem that was fixed at four.

Prepare contact lists, familiar communication channels and message templates before the incident. The workbook recommends deciding who drafts, reviews, approves and releases external updates. Templates should leave room for the known impact, uncertainty, current action and next update time; preparation reduces the work that must be improvised under pressure.

Different audiences need different detail. Responders need current state and task ownership; internal stakeholders need impact, uncertainty and decisions required; customers need to know whether they are affected and what to do. A shared factual update can support all three, with additional technical or decision detail routed to the appropriate audience. This audience split is a practical design choice rather than a prescribed Google format.

After: the document is not the deliverable

A postmortem records the incident, its impact, mitigation or resolution actions, contributing causes, and follow-up work to prevent recurrence. It gives the organization a formal way to learn; without that mechanism the same failure can continue recurring.

The SRE book lists common postmortem triggers: user-visible degradation beyond an agreed threshold, data loss, on-call intervention such as rollback or rerouting, prolonged resolution, and monitoring failure. Teams agree their criteria in advance and retain flexibility; any stakeholder can also request a review.

An intervention that successfully prevents user harm can still reveal a weakness worth investigating. A customer discovering a problem first is also a reason to review detection, but it is not automatically a monitoring failure. Compare the event with the intended coverage, detection delay and impact threshold. The customer may have reported within that interval, or the event may intentionally use a nonpaging response. Record whether the agreed detection obligation was missed and whether the policy itself needs to change.

Blameless is a mechanism, not a courtesy

Google’s blameless postmortem approach investigates contributing conditions without indicting a person or team. It starts from the assumption that participants acted in good faith with the information available, then asks what made the action seem reasonable and which safeguards were missing.

Punishing disclosure can discourage people from reporting mistakes and near misses, leaving the organization with less evidence from which to learn. The SRE book connects blameless practice to safety work in healthcare and aviation. The operational objective is candid reporting and a deeper investigation, not a claim that every decision was correct.

Follow-up should improve the systems and processes that support decisions. A request for greater care leaves the same confusing signals or unsafe path available to the next responder.

Blameless analysis must still identify where services can improve. The workbook describes a postmortem that stopped at an engineer bypassing automation and manually turning up a cluster, which triggered a preexisting bug. Recording that sequence can be accurate while leaving the causal investigation incomplete: examine the automation that allowed the manual path and the defect it exposed. Blameless postmortem explains how to investigate those conditions without substituting personal blame for system improvement.

Action items are where postmortems fail

The workbook illustrates several ways action items can fail to produce useful changes. Review the proposed work against these failure modes rather than treating the document itself as the outcome.

• Only mitigations. Faster recovery can reduce harm without removing the initiating defect. Include prevention where it is feasible and justified, and state whether each action targets recurrence, impact, detection or recovery.
• Relying on greater care. Replace a request to avoid mistakes with specific system or process changes that support safer choices. Training can still help, but should not substitute for an available guardrail against the same error.
• No execution order. Equal priority labels do not establish what should happen first. Sequence actions by risk, dependency, effort and urgency, with an owner for resolving conflicts.
• No completion criterion. Replace vague improvement goals with observable outcomes and verification steps, so the owner and reviewer can decide whether the change achieved its purpose.
• No tracking. Put the agreed work in a system where owners, deadlines and progress are reviewed. An item recorded only in the postmortem can otherwise be forgotten.

The workbook quotes a Google policy requiring at least one high-priority tracked action for a user-affecting postmortem, with explicit review of exceptions. It illustrates how an organization can require follow-through; local priorities and exception authority still need to be defined.

The workbook records a Google policy requiring a tracked P0 or P1 action for each postmortem following user harm, with rare exceptions reviewed by the responsible leader. Its purpose is to make follow-through enforceable: users gain no prevention benefit from a document whose actions remain undone.

Balance recognition for documenting incidents with recognition for implementing their action plans. Rewarding only completed postmortems can leave an expanding backlog of unfinished changes. Documentation quality and demonstrated improvement need separate completion checks.

Review the postmortem for evidence completeness, impact, contributing causes, action suitability and priority, and distribution to the people who can use the lessons. The SRE book treats this review as part of completing the document, not as a substitute for implementing and checking its actions.

The workbook describes a rack decommission in which an automation bug and inadequate rate limits took thousands of serving machines offline. In a similar incident three years later, the implemented actions limited the rate and extent of the failure. That later event demonstrated reduced blast radius; it does not establish that the controls provided no benefit before then. Assess each action against the recurrence or impact it is intended to reduce.

On-call is a budget, and it is spent in people

Having someone available to respond is a staffing and scheduling obligation. The SRE book distinguishes the share of time engineers spend on call from the incident load during each shift. Scheduled coverage and the work generated while covering it need separate budgets.

Google’s example reserves at least half of SRE time for engineering, limits on-call allocation to a quarter and allows other operational work in the remaining quarter. Its continuously staffed primary-and-secondary example yields eight engineers in one site. Use the same reference period for coverage demand and each person’s allocation. If H is the number of hours requiring coverage, c the simultaneous on-call positions, T the reference hours per person, and f the allowed fraction of those hours on call, the coverage-only minimum is ceil(H × c / (T × f)). For a 168-hour week, two continuously staffed positions and a 25% calendar-time allocation, ceil(168 × 2 / (168 × 0.25)) = 8. This measures scheduled availability, not 25% of a 40-hour working week. Add allowance for leave, training and handoffs, and assess actual incident work separately. Shared secondaries or business-hours coverage change the inputs. Google’s dual-site minimum of six engineers per site also includes a judgment about maintaining sufficient team size; it is not uniquely derived from this ratio.

The SRE book links night shifts to adverse health effects and describes multi-site follow-the-sun coverage as a way to avoid them. Keep each rotation small enough that engineers remain familiar with production: infrequent duty can create operational underload. Account for the additional communication and coordination work across sites.

Google reports an average of six hours for an incident’s analysis, remediation and follow-up work together, and uses that assumption to propose at most two incidents per 12-hour on-call shift. It also considers the distribution over time and the pressure created by recurring pages. These are planning assumptions to compare with the local mix of incidents, not universal limits or a requirement to wait a quarter before responding to overload.

Under the six-hour assumption, three incidents per shift generate eighteen hours of total work. If only twelve hours of work capacity are available, sustained arrivals at that rate create a backlog. Follow-up may be assigned to another engineer or a later scheduled period, but that capacity must be budgeted rather than assumed. Track which work is deferred and how long it remains open. Alert fatigue describes the additional cost of repeated interruptions.

On-call compensation can include time off or cash, with cash limited to a stated share of salary. In the SRE book’s model, that cap also limits the on-call work allocated to an individual. It expresses the staffing load in financial terms that can support a capacity decision as well as fair compensation.

The book also describes how stress can favor rapid, familiar responses over deliberate examination during complex outages. Prepared procedures reduce the number of coordination decisions responders must invent under pressure, while escalation brings in help when the current team is overloaded.

A repeated alert illustrates the risk: after three incidents caused by one dependency, a responder may assume the fourth has the same cause without checking. It may or may not. Verify the current evidence while using prior incidents as hypotheses. How to arrange shifts, secondary coverage, escalation and handoff is developed in on-call rotation.

What to build before the next one

Prepare the incident-management framework before the pager fires. Once an outage begins, responders need familiar authority, communication and escalation paths rather than having to design them under pressure.

A small team can start with a manageable part of the framework: let on-call responders delegate and escalate, prioritize safe mitigation, and define command, communications and operations responsibilities. Write declaration criteria, prepare a contact list and channel, and create message templates for acknowledging and ending the response. Agree postmortem triggers and require an owned, prioritized, tracked action for user-affecting cases. Plan the work against the team’s capacity; completing it this month is a local choice rather than a universal delivery promise.

Rehearse because incident-management skills decline without use. The SRE workbook suggests using the framework for ordinary coordinated changes across teams or time zones, and reenacting past incidents from their postmortems. Engineers play the recorded roles, with the original commander helping the exercise reflect the actual incident.

During a response, notice your emotional state and seek more support if you feel panicky or overwhelmed. Rotate responsibilities in exercises and incidents: someone who commanded last time can take another role. If only one person can command, the team has a single point of failure that infrastructure redundancy cannot address.

The opening shift failed because the engineer had to coordinate while doing the technical work, communication had no owner, and an outside production change was uncoordinated. Named command, communications and operations responsibilities, written declaration criteria and a prepared message template would have addressed those response failures alongside the original technical fault.

Review both the response and the follow-through. During the incident, responders should know who is commanding and who may change the system. Afterwards, distinguish an action that is registered or scheduled from one that is implemented and verified. Track owners, deadlines, overdue reviews and any accepted residual risk. A completion check or targeted exercise should show whether the intended prevention or impact reduction works; a ticket’s existence alone cannot establish that.

References

• “Managing Incidents,” Google SRE Book
• Google SRE Book — “Effective Troubleshooting”
• Google SRE Book — “Postmortem Culture: Learning from Failure”
• Google SRE Book — “Being On-Call”
• “Incident Response,” Google SRE Workbook
• Google SRE Workbook — “Postmortem Culture”
• FEMA, ICS Review Document (March 2018), p. 5