Incident Commander

The incident commander holds the overall state of an incident and coordinates who does what. Software incident management adapts the Incident Command System, whose development FEMA traces to interagency FIRESCOPE work in the 1970s. Google uses its principles of explicit responsibilities and scalable delegation for software response.

In Google’s model, the commander sets priorities, assigns responsibilities and retains the roles that have not yet been delegated. This allows the response to start with one person and expand as more help becomes necessary.

What the role is for

The SRE workbook groups the role around coordination, communication and control: assign the work, keep participants and stakeholders informed, and maintain a shared view of the response. These goals help identify coordination gaps without prescribing a fixed number of people.

Google’s operations and communications leads report to the incident commander. Operations coordinates technical response; communications provides updates and handles inquiries. Planning supports the longer-running needs of the response, including staffing, handoffs and records of emergency changes.

Production changes should be coordinated through operations. The commander keeps that responsibility visible and helps prevent uncoordinated interventions; access controls, change records and responder discipline support the same boundary.

Choosing the commander and the technical lead

Choose the commander for their ability to coordinate the response, and choose operations staff for the technical work required. Google’s separation of responsibilities lets each role act within its assigned scope without repeatedly second-guessing the other.

The deepest technical expert may be most useful in operations when no one else can diagnose the affected system. In another incident, that person may be well placed to command while other engineers investigate. The practical constraint is attention: if debugging prevents the commander from tracking state and coordinating decisions, explicitly delegate operations or hand over command. Expertise is not a disqualification for command.

Practice coordination as well as technical response. Google recommends rotating roles during exercises so that more than one person can command, communicate or lead operations. This reduces dependence on one responder without requiring every engineer to have identical expertise.

It scales down to one person

The most common objection is that a small team cannot staff four roles, and the framework answers it in a single clause: “De facto, the commander holds all positions that they have not delegated.” The workbook repeats it — “by default, the IC assumes all roles that have not been delegated yet.”

So an incident handled by one on-call engineer already has a commander: them. The roles are not seats to be filled but responsibilities to hand off as the incident grows, and typically “the person who declares the incident typically steps into the IC role.” Growth and shrinkage are both provided for: an overloaded lead should “ask the planning lead for more staff” and may delegate by creating subincidents; and when things calm down, “if the incident becomes small enough, the CL role can be subsumed back into the IC role.”

There is also a documented move for the case where the commander turns out to be the person who should be fixing things: “the IC may either hand off their role to someone else and assume the OL role, or assign the OL role to someone else.” Changing your own role deliberately is part of the framework; drifting into two without saying so is not — the problem is the silence, since holding two is exactly what the default prescribes until they are handed out.

What the commander does not do

The negative list is where most of the practical value is. Read it as what the commander stops doing the moment there is somebody to hand it to — not as a prohibition that would contradict the default above, where an undelegated role is still theirs.

• Not making the changes, once operations is staffed. Modifying the system belongs to that role; while it is undelegated the commander holds it, and the first spare responder is the reason to give it away.
• Not debugging, once anyone else can. Holding the overall state and reading a stack trace compete for the same attention, so the commander who finds themselves deep in one has lost track of the other — which is the documented cue to hand over command and take the operations role instead.
• Not the senior person present. Command is delegated for the duration and is not a statement about rank; an executive who arrives does not become commander by arriving.
• Not the one who writes the postmortem alone. Command ends when the incident does; the learning is a separate, shared piece of work.

What they do own, besides coordination, is the record: “The incident commander’s most important responsibility is to keep a living incident document.” And they own removing obstacles — the commander “can remove roadblocks that prevent Ops from working most effectively,” which in practice means approvals, access and getting the right person woken up. A commander with no authority to do those things has the title and not the role.

One tooling caution attaches to the document and generalizes: “depending on the software you are trying to fix as part of your incident management system is unlikely to end well.” If your incident document lives on the platform that is down, you have discovered this the hard way.

The handoff is deliberately theatrical

Long incidents outlive shifts, and the framework is emphatic that command must be transferred explicitly: “It’s essential that the post of incident commander be clearly handed off at the end of the working day.”

the outgoing commander should be explicit in their handoff, specifically stating, “You’re now the incident commander, okay?“, and should not leave the call until receiving firm acknowledgment of handoff. The handoff should be communicated to others working on the incident so that it’s clear who is leading the incident management efforts at all times.

The formality looks excessive until you notice what it prevents. Two people each assuming the other is in command produces behavior indistinguishable from nobody being in command — questions unanswered, decisions unmade, everyone waiting — and it is discovered only when something needs deciding. A verbal acknowledgment is a cheap check on an expensive silent failure. The same reasoning covers the announcement to everyone else: responders need to know whom to ask, and a handoff nobody heard has not happened.

Adopting the role is a small change. Say that the person who declares an incident is the commander until they hand it over, name the other two roles so there is something to delegate into, and rehearse the handoff sentence once so it does not feel absurd the first time. How declaration, communication and the work after an incident fit around it is worked through in Fly the Airplane First.

References

• “Managing Incidents,” Google SRE Book
• “Incident Response,” Google SRE Workbook
• FEMA, ICS Review Document (March 2018), p. 5