On-Call Rotation

An on-call rotation assigns people to be reachable for production problems during defined hours. It must provide response coverage while leaving enough capacity for engineering and recovery from interruptions. The staffing and incident-effort figures below come from Google’s operating model; measure and adapt them to the service being supported.

Google separates scheduled on-call allocation from the incident workload arriving during those shifts. Management must balance both: a calendar can have every slot filled while the people assigned cannot absorb the work.

Quantity: the ceiling implies a team size

Google’s model reserves at least half of SRE time for engineering, caps on-call allocation at a quarter and allows other operational work in the remaining quarter. Treat scheduled availability and hours actually spent handling incidents as separate measurements.

For a single-site rotation with a primary and secondary continuously assigned, Google gives a minimum of eight engineers. With week-long assignments, each person carries one of the two roles for roughly one week in four. Its dual-site suggestion of six per site also reflects a judgment about sufficient team size, rather than just the coverage fraction.

Use the same reference period for coverage demand and each person’s allocation. If H is the number of hours requiring coverage, c the simultaneous on-call positions, T the reference hours per person, and f the allowed fraction of those hours on call, the coverage-only minimum is ceil(H × c / (T × f)). For a 168-hour week, two continuously staffed positions and a 25% calendar-time allocation, ceil(168 × 2 / (168 × 0.25)) = 8. This measures scheduled availability, not 25% of a 40-hour working week. Add allowance for leave, training and handoffs, and assess actual incident work separately. Shared secondaries or business-hours coverage change the inputs. Google’s dual-site minimum of six engineers per site also includes a judgment about maintaining sufficient team size; it is not uniquely derived from this ratio.

Spreading across sites buys two things and costs one.

• No night shifts. “Night shifts have detrimental effects on people’s health,” and “a multi-site ‘follow the sun’ rotation allows teams to avoid night shifts altogether.”
• A rotation that is not too large. The less obvious benefit — “limiting the number of engineers in the on-call rotation ensures that engineers do not lose touch with the production systems.” Being on call twice a year is its own failure mode, discussed below.
• Coordination overhead. “However, multi-site teams incur communication and coordination overhead,” so the choice “should be based upon the trade-offs each option entails, the importance of the system, and the workload each system generates.”

The secondary rotation is more flexible than it looks. It may be “a fall-through for the pages the primary on-call misses,” or a split of duties where “the primary on-call handles only pages, while the secondary handles all other non-urgent production activities.” And where a dedicated secondary is hard to staff, “it is common for two related teams to serve as secondary on-call for each other,” which “eliminates the need for an exclusive secondary on-call rotation” — often the most practical option for a small organization.

A shift needs an acknowledged owner

Consider a hypothetical rotation changing primary at 09:00. Before that time, the outgoing responder shares active incidents, temporary changes and their expiry times, pending work, relevant runbooks, and any weakened recovery options. The incoming responder checks access and paging delivery, then explicitly accepts responsibility. Record the transfer in the shared channel and confirm that the routing schedule now points to the incoming primary.

Set the expected acknowledgment and escalation intervals from the service’s impact and response needs. If the primary does not acknowledge, route to the secondary and then a named backup or duty manager; responders must also be able to request help before those timers expire. Specify how backup coverage is arranged for leave, illness and simultaneous incidents. If the next primary cannot accept a shift, activate that fallback rather than letting responsibility disappear when the calendar changes.

An ongoing incident may retain its commander while pager ownership changes, or may transfer both roles. State which responsibilities moved and obtain acknowledgment for each. The outgoing responder remains responsible until the agreed handoff or backup procedure establishes a new owner; the policy must provide relief rather than depending on indefinite personal availability. Exercise failed acknowledgments and unavailable backups as well as an ordinary shift change. This example is a proposed operating procedure, applying the SRE guidance on explicit handoffs and escalation.

Incident workload and follow-up capacity

The second axis has a measured number behind it, and it is the one that converts pages into headcount. A shift must leave “sufficient time to deal with any incidents and follow-up activities such as writing postmortems.” An incident here means “a sequence of events and alerts that are related to the same root cause and would be discussed as part of the same postmortem” — so a flurry of related alerts is one incident, not nine.

We’ve found that on average, dealing with the tasks involved in an on-call incident—root-cause analysis, remediation, and follow-up activities like writing a postmortem and fixing bugs—takes 6 hours. It follows that the maximum number of incidents per day is 2 per 12-hour on-call shift.

At that measured average, three incidents per shift generate eighteen hours of total work, including response and follow-up. A sustained arrival rate of that size exceeds a single twelve-hour work allocation. Some follow-up can occur later or be assigned to other engineers, but its capacity and deadlines must be explicit. Track backlog and deferred improvements alongside urgent response so that present incidents do not consume all the capacity needed to reduce future ones.

Two further consequences come from the same place. The target distribution is “very flat over time, with a likely median value of 0” — a typical shift should be quiet, which is a much stronger claim than an acceptable average. And a component that pages daily is a leading indicator rather than a nuisance: “it is likely that something else will break at some point, thus causing more incidents than should be permitted.” When the limit is exceeded for a quarter, the documented response is structural — “corrective measures should be put in place to make sure that the operational load returns to a sustainable state” — not a rota reshuffle.

Compensation is a control, not only fairness

Out-of-hours work is compensated “time-off-in-lieu or straight cash compensation, capped at some proportion of overall salary,” and the reason the cap exists is the interesting part: “the compensation cap represents, in practice, a limit on the amount of on-call work that will be taken on by any individual.” The stated effect is two-sided — it keeps people willing to take shifts while it “promotes a balanced on-call work distribution and limits potential drawbacks of excessive on-call work, such as burnout or inadequate time for project work.”

Structurally, a cap converts on-call load into a number that appears in a budget. Arguments about sustainability tend to lose to arguments about delivery dates, because one side has figures. A compensation line is the same load expressed in the currency the rest of the organization already uses, and it makes an unsustainable rotation expensive before it becomes a resignation.

Why quiet matters more than heroism

Stress can favor rapid, familiar responses over deliberate examination during a complex outage. The SRE book uses this distinction to motivate preparation, support and escalation, not to imply that responders lose all ability to reason.

Reduce avoidable pressure by providing clear roles, accessible procedures and relief when a responder is overloaded. These controls protect both service decisions and the people making them.

The book illustrates confirmation bias with a repeated alert: after three occurrences traced to one dependency, responders may automatically attribute the fourth to it. The fourth could have the same cause or a different one. Check the current evidence rather than allowing familiarity to replace diagnosis.

So a sustainable rotation is a reliability control and not a perk. It also needs stated permissions, because escalating feels like failing: tell on-call engineers explicitly that “they can delegate and escalate during an incident,” and make self-assessment part of the practice — “if you start to feel panicky or overwhelmed, solicit more support.”

The opposite risk is real too. A rotation so large or so quiet that engineers “lose touch with the production systems” produces responders who have never used the tools under pressure. The answer is not more incidents but rehearsal — practicing the response, and rotating roles so that competence is spread rather than concentrated.

Review scheduled coverage, actual response work, follow-up backlog and the distribution of incidents per shift. The mean describes average work to absorb, while peak shifts reveal overload hidden by that average; nine quiet shifts and one with twenty incidents still have a median of zero. Measure local effort instead of importing six hours as a constant. Also verify that escalation and handoff work when a responder is absent. How an active incident is commanded is developed in Fly the Airplane First; the notification burden is discussed in alert fatigue.

References

• “Being On-Call,” Google SRE Book
• Google SRE Book — “Managing Incidents”
• “Incident Response,” Google SRE Workbook