Toil

Toil is recurring operational work associated with running a production service. Google’s SRE book describes six tendencies: manual effort, repetition, automatable work, reactive handling, little enduring value and growth with the service. Use them together as diagnostic clues, not a rigid all-or-nothing checklist.

Toil is not defined by whether someone dislikes a task. Administrative overhead is a different category, and a difficult one-time improvement can produce lasting value. Classify the work by what it does and why it repeats.

The six attributes, as questions

AttributeThe test
ManualHow much hands-on work is required, separate from machine elapsed time?
RepetitiveIs the same operational task recurring, rather than a novel problem?
AutomatableCould automation or redesign remove the need, including repeated judgment caused by avoidable complexity?
TacticalIs the work repeatedly reacting to operational events?
No enduring valueDoes the task leave the system essentially unchanged?
Scales with growthDoes recurring effort increase with service size or demand?

Growth in repeated work is a useful warning signal, but doubling with traffic is neither a complete definition nor the only reason to investigate. Consider the task’s repetition, necessity and lasting result alongside its scaling behavior.

Novel diagnosis and design can be engineering even when reactive. Repeated judgment does not automatically exempt a task from toil: Google’s footnote describes complex recurring alerts whose need could be removed by redesign. Separate the genuinely new decision from the repetitive work the system keeps generating.

Why a ceiling exists

The SRE introduction explains that recurrent manual operations can require more staff as services grow. Engineering that removes the work can improve that relationship and preserve capacity for further changes.

Google’s SRE model describes a 50% aggregate upper bound on operations and an aim to preserve at least half of each SRE’s time for engineering over longer periods. These are organizational targets with different aggregation scopes, not a requirement that every week have the same split. Define what your own measurements count and their observation period.

Some toil is unavoidable and can be acceptable in limited amounts. Persistent overload can displace engineering, reduce satisfaction and encourage more work to be routed to the same team. Look at individual load as well as team averages when deciding whether the arrangement is sustainable.

Google illustrates potential on-call exposure with one primary and one secondary week in a six-person rotation: two of six weeks, or about 33%; an eight-person rotation gives 25%. That is a duty-allocation example, not proof that 33% of hands-on time was spent on toil before any ticket arrived. Measure actual work and engineering interruptions under the local rota as well as scheduled duty share.

Making an operational limit actionable

Google’s model redirects excess operational work to product development through ticket reassignment and participation in on-call rotations. It depends on organizational agreement and receiving teams able to do the work. A target alone does not create a working transfer mechanism.

Agree who can receive which work before overload occurs. Preserve existing service response obligations while managers adjust scope, capacity or ownership. A runbook cannot substitute for the recipient’s authority, skills and available capacity.

Four responses to recurring work

Consider four responses according to total effort, risk and maintenance cost. They can be combined; their order here is not a ranking.

  1. Eliminate the need. A design change can remove a recurring task altogether. Compare the cost and risk of that change with the expected work avoided.
  2. Let the requester serve themselves. The work does not disappear, but it stops being yours and stops being a queue. This is the whole premise of self-service on an internal platform — and the reason to be careful with the accounting. Handing a task to the person who wanted it done removes a handoff and the waiting around it, which is real; it does not by itself remove the work, and if the requester now spends longer on it than you did, the organization is worse off while your toil number improves. The honest form of the claim names both sides: what the request used to cost you, and what the self-service path costs the person who now walks it.
  3. Automate it. Software can remove repetitive handling, but it introduces code to maintain and diagnose. Evaluate its lifecycle cost and the work that remains.
  4. Reassign responsibility where appropriate. Feedback to the owning team can motivate a fix, including automation. Reassignment and better tooling are compatible, and moving the work is justified only with an agreed recipient and support arrangement.

Platform teams can accumulate recurring requests that crowd out capability development. DORA describes this reactive ticket-handling failure mode. Examine the manual effort each request takes, how fast the volume grows, and how much time is left for the improvements that could reduce it.

Measure recurring work and choose how to reduce or manage it. Some work can be eliminated, some automated and some retained or reassigned within an agreed service contract. The Load Has to Land Somewhere applies this to platform responsibility; cognitive load describes the related question of what a team must understand to perform its work.

References


Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Similar Posts

Questions, corrections, or additional insights?

This site uses Akismet to reduce spam. Learn how your comment data is processed.