Blameless Postmortem

A postmortem records an incident’s impact, response, contributing causes and follow-up work. In Google’s blameless approach, the investigation examines actions and system conditions without indicting a person or team, starting from the assumption that people acted in good faith with the information available.

The SRE book treats writing a postmortem as a learning opportunity. Documenting what happened gives teams evidence for reducing recurrence or impact; the document must lead to reviewed, tracked work to achieve that purpose.

Blameless is a mechanism, not a courtesy

Blameless practice supports disclosure. If reporting an error invites punishment, people may withhold information that an investigation needs. A review should make it possible to describe actions precisely while examining the conditions in which they were taken.

Google warns that blame can drive issues out of view, including when frequent postmortems are treated as evidence against a person or team. Interpret report counts alongside exposure, incident severity and reporting behavior; fewer reports alone do not establish fewer failures.

The SRE book links this practice to safety work in healthcare and aviation. Its practical emphasis is on changing systems and processes to support better choices rather than ending the investigation at a human error.

Blameless does not mean saying nothing

Blameless writing should still identify specific weaknesses and explain how the service can improve. Avoiding accusations does not mean omitting the actions, evidence or decisions that contributed to the event.

The clearest illustration of the line is a bad postmortem whose stated root cause was that a named engineer “ignored the automation setup and ran the cluster turnup logic manually, which triggered an existing bug.” Notice that every clause there may be factually accurate. It is still the wrong answer to why did this happen, and not because it is unkind: it stops at the last human in the chain. Unexamined are why the manual path existed, why it was easier than the automated one, why nothing prevented it, and the “existing bug” that the sentence mentions and then walks past. A blameless version of the same investigation goes further into the system, not less far.

Language does real work here, because “blameful language stifles collaboration between teams.” The documented example is a question that sounds reasonable and lands badly — asking a manager “why aren’t you making sure that everyone finishes the training?” will “instantly put the recipient on the defensive,” where stating what the postmortem showed and proposing a requirement gets the same point into the action items instead of into an argument.

When to write one

The SRE book recommends agreeing postmortem criteria before an incident. Its common triggers include the following, with room for teams to adapt thresholds and for any stakeholder to request a review.

• “User-visible downtime or degradation beyond a certain threshold“
• “Data loss of any kind“
• “On-call engineer intervention (release rollback, rerouting of traffic, etc.)“
• “A resolution time above some threshold“
• “A monitoring failure (which usually implies manual incident discovery)“

A rollback that prevents visible harm can still reveal a weakness worth investigating. When a customer reports an outage first, compare the event with the intended detection coverage, delay and severity policy before calling it a monitoring failure. A report within the expected detection interval does not itself show a failure. Teams may nevertheless choose to review every customer-reported incident as an explicit local policy.

Where postmortems actually fail: the action items

A well-written, widely shared, entirely blameless postmortem can still change nothing. The policy quoted from Google’s vice president for 24/7 operations is the sharpest statement of the standard:

To our users, a postmortem without subsequent action is indistinguishable from no postmortem. Therefore, all postmortems which follow a user-affecting outage must have at least one P[01] bug associated with them. I personally review exceptions. There are very few exceptions.

The workbook’s deliberately bad example illustrates several failure modes in action plans. The order below is an editorial grouping, not a measured ranking of how often each causes failure.

1. Nothing is tracked. Assign the work to an owner and review its deadline and progress in a tracking system. A document-only item is easy to lose after the response ends.
2. Everything is mitigative. Faster recovery reduces harm but may leave the initiating defect intact. Consider preventive fixes as well, and make clear whether each action targets recurrence, impact, detection or recovery.
3. It relies on greater care. Prefer a specific guardrail or process improvement over an instruction to make fewer mistakes. Training can support that change, but should not replace feasible protections against the same error.
4. The verbs cannot be finished. Define an observable outcome and a verification method for each improvement so its owner and reviewer can assess completion.
5. Everything has the same priority. Establish an execution order using risk, dependencies, effort and urgency. Equal labels alone neither provide that order nor prove that nothing will be done.

The incentive trap behind all five is named directly: “If you reward engineers for writing postmortems, but not for closing the associated action items, you risk an unvirtuous cycle of unclosed postmortems. Ensure that incentives are balanced between writing the postmortem and successfully implementing its action plan.” A team can become genuinely excellent at postmortems and remain exactly as unreliable as it was.

Review evidence, impact, contributing causes, action suitability and priority, and whether the lessons reached relevant stakeholders. Keep document review separate from action completion: an assigned or scheduled fix is still pending, and an implemented fix needs verification against its intended outcome. Review overdue items and explicitly accept or reduce remaining risk.

Evidence that it pays, three years later

The claim behind all of this is stated plainly — “our experience shows that a truly blameless postmortem culture results in more reliable systems” — and the workbook’s case study is the closest thing to a measurement. A routine rack decommission hit a bug in maintenance automation which, “combined with insufficient rate limits, caused thousands of servers carrying production traffic to simultaneously go offline.”

Then: “Three years after this outage, we experienced a similar incident” — satellites drained again, latency up again — and “the action items implemented from the original postmortem dramatically reduced the blast radius and rate of the second incident.”

The later incident demonstrates reduced impact from the implemented changes. It does not show that their benefit began only three years after implementation. Preventive controls can reduce exposure before another incident tests them; targeted exercises and completion checks can provide evidence sooner. Track both implementation and the intended reliability outcome.

Starting is cheap and the sources are direct about not overbuilding: introducing this is “as much a cultural change as it is a technical one,” best begun with “a very basic postmortem procedure” that you then tune, since “there is no one size that fits all.” Write down the triggers, require one tracked and prioritized action item for anything user-affecting, and have someone senior review the first few — ideally leaders who “consistently exemplify blameless behavior,” because the tone is set by whoever asks the first question in the review. How this fits with incident command and the work during an outage is in Fly the Airplane First.

References

• “Postmortem Culture: Learning from Failure,” Google SRE Book
• “Postmortem Culture,” Google SRE Workbook