Capacity Planning and Handling Overload

The SRE workbook illustrates the problem with Dressy, a fictional retailer running in three equally sized regions. One morning each region served 90 requests per second. By 11:10, region A received 400 while B and C had dropped to 40, and most requests to A returned 503 errors. The load balancer settled into that state and considered its job done.

The components followed their configured rules. Earlier that week the team enabled load shedding: above a CPU threshold, a server rejects work rather than processing it. Region A crossed first, rejecting 10%, then 20%, then 50% while CPU stayed flat. The utilization-aware balancer divided CPU by incoming requests and calculated one-tenth the CPU cost per request in A compared with B and C. That misleading appearance of efficiency attracted still more traffic.

The balancer and the shedding policy produced an outage while following their own rules. Their interaction made rejection look like efficient processing. Understanding overload therefore requires connecting the measurement of demand to capacity decisions and to the work that will be dropped when capacity is insufficient.

Relate request rate to resource demand

Google’s SRE book warns against treating aggregate queries per second as a stable proxy for resource demand. A load test can establish useful QPS capacity when it specifies the request mix, software version, duration, and acceptable latency and error rate. The number becomes unreliable when those conditions change without revalidation.

Requests differ in cost. The client can change which operations it asks for, the traffic mix can vary through the day, and a new release can make a common operation cheaper or more expensive. A balancer that assumes a fixed resource cost per request can therefore make a poor placement decision even when its request counts are accurate.

The SRE book instead models available resources directly. For a service allocated 500 CPU cores and a terabyte of memory in a datacenter, the plan must account for demand against those resources. CPU time consumed by each request provides one measure of its cost; request counts remain useful for describing demand and throughput.

Google reports that CPU consumption worked as a provisioning signal for many of the services discussed. Garbage collection can translate memory pressure into CPU work, and other resources can sometimes be provisioned with enough margin that CPU becomes the first constraint. This is an implementation choice, not a rule that memory, I/O, connections, or concurrency are always secondary. Account for them separately when they can become bottlenecks.

Pair request-rate observations with the resources and concurrency that constrain the service. Retain the workload and acceptance conditions behind each capacity measurement. A changed caller mix or hot-path implementation is a reason to revalidate that measurement; it does not prove that the old number has already become wrong.

Saturation arrives before the resource runs out

Resource pressure can damage service before utilization reaches 100%. As CPU becomes scarce, requests slow down and trigger a chain of additional demand, described in the SRE book:

More requests in flight. Slower handling keeps more requests active concurrently, increasing memory, thread, file-descriptor and backend-resource demand.
Queues saturate. Sustained arrivals above processing capacity fill waiting queues, adding both latency and memory demand.
Deadlines start being missed. Slow responses can outlast the caller’s waiting limit and trigger retries. For a read whose late answer is no longer useful, that can waste work. An accepted write or durable job may still need to complete or record its outcome; a canceled wait does not itself cancel the operation.
Health checks fail, and restarts remove capacity. Starved threads can delay health-check responses. A scheduler may then restart tasks that are busy rather than broken, reducing capacity during the shortage.

Memory pressure can reinforce CPU exhaustion: slower requests retain more memory, garbage collection consumes more CPU, and still less CPU remains to process requests. The SRE book calls this reinforcing cycle the GC death spiral.

Degradation can generate additional load rather than ending in a smooth decline. Timed-out requests retry, busy tasks restart, and reduced memory causes more cache misses just as available capacity falls. These secondary symptoms complicate diagnosis because they can resemble the initiating fault.

Latency at a high percentile can provide early evidence of saturation, especially when queueing increases before an aggregate utilization threshold is crossed. It is not guaranteed to lead every resource alarm, and latency can rise for other reasons. Interpret it alongside resource and queue measurements; Only a Few Times a Day develops those signals. Propagation across service boundaries is described in cascading failure.

The queue is where your latency is hiding

Thread-per-request servers commonly place a waiting queue before the worker pool and reject arrivals when that queue is full. Queue capacity therefore changes both the latency of admitted requests and when excess work is refused.

With perfectly constant arrivals and service times, enough workers can handle each arrival without queueing. The SRE book uses that idealization to connect queue growth to sustained demand above capacity. Real arrivals and service times vary, so retain the distinction between that ideal case and observed queueing. A stable queue — one whose server is faster than its arrivals on average — can still have work waiting most of the time once utilization is high, without ever growing without bound. How often depends on how variable the arrivals and service times are, and the standard textbook case gives a number: for a single server with Poisson arrivals, independent exponential service times and unbounded queue space, the probability that somebody is waiting at a random moment is ρ², which is about 90% at 95% utilization. Regularity changes that completely — arrivals exactly one second apart served in exactly 0.95 seconds reach the same utilization with nobody ever waiting — so the figure is a property of the variability, not of the utilization alone. So a non-empty queue is not by itself evidence of insufficient capacity. What distinguishes a buffer from a backlog is the trend and the age: whether arrivals exceed completions over a window rather than within a burst, whether the oldest item in the queue keeps getting older, and whether requests are missing their deadlines. A queue that grows without bound is a system that cannot keep up, storing the evidence.

The SRE book illustrates the queueing cost with 100 ms of service time and a queue ten times the worker count. Under a simplified saturated-pool approximation, waiting is about the number of requests ahead divided by the number of workers, multiplied by service time. Ten waves of work add roughly one second before the request’s own 0.1 seconds: about 1.1 seconds total. This is eleven times the no-wait service time, not an elevenfold consequence of increasing a queue setting tenfold. At one wave the corresponding approximation is 0.2 seconds, so increasing it to ten waves changes that comparison by 5.5 times. These estimates concern an accepted request near the back of an occupied queue; a new arrival to a full bounded queue may instead be rejected. Worker progress, service-time variation, and actual occupancy determine observed latency.

For steady traffic, the SRE book favors a queue small relative to the worker pool, such as half its size or less, to reject overload before long waits develop. Gmail’s example uses no waiting queue and sends work to another task when all workers are occupied. Bursty demand can justify a buffer sized from worker availability, processing time, and burst size and frequency. Early rejection protects the users whose work the service can still complete; choose the buffer from the actual arrival pattern.

The SRE book also discusses LIFO and controlled-delay queueing to avoid spending scarce capacity on stale work. Its example is a search request whose user has already refreshed the page. Propagated deadlines help identify reads whose late responses can be discarded. That policy requires an operation contract: an accepted payment or durable asynchronous job may need completion, outcome recording, cleanup, or compensation after the caller stops waiting. Cancellation must be observed by the application and does not undo committed effects. Decide which queued operations can be abandoned before applying a newest-first or expiration policy.

Capacity planning, and what it cannot buy

Provisioning begins with the demand and failure capacity the service must carry. In the traditional cycle described by the SRE book, teams collect demand forecasts, devise build and allocation plans, obtain review and approval, then deploy and configure. They repeat that cycle as assumptions change, deployments slip and budgets are cut; a signed-off plan is not a permanent capacity decision.

Google describes conventional capacity planning as brittle and laborious. A change in service efficiency, adoption, delivery dates, or performance targets can force a wider allocation review. Gathering forecasts is slow and error-prone, and resources in one location cannot always satisfy another location’s latency constraints. Allocating these constrained demands is a bin-packing problem, an NP-hard problem in the general formulation discussed in the book. Manual spreadsheets add risks from stale data and difficult change tracking.

The SRE book’s example uses evenly spread peak demand of 19,000 QPS and a measured breaking point of 5,000 QPS per cluster. Rounding 19,000 / 5,000 up gives four; adding two gives six clusters for N+2. All six may serve traffic in normal operation at about 3,167 QPS each, with four survivors carrying 4,750 each after two failures. N+2 does not require two idle clusters. Nor does a breaking point establish the load at which latency and errors still meet the objective: if that verified operating capacity were 4,000 QPS, the same simple calculation would require seven clusters. Validate the failure domains, redistribution, startup time, and workload assumptions before setting a reliability target from this count.

Express capacity requirements programmatically so a plan can be regenerated when demand or assumptions change. Google’s abstraction example moves from a specific resource placement to demand, redundancy and the reliability outcome that justify the placement:

Reserve 50 cores in clusters X, Y and Z for Foo. The placement is explicit, but its reason remains unstated.
Give Foo a 50-core footprint in any three clusters in region YYY. This permits placement choices without yet explaining the requirement.
Meet Foo’s demand in every geographic region with N+2 redundancy. The failure-capacity requirement now explains the allocation.
Operate Foo at five nines of reliability. This states the desired outcome and the consequence of missing it; N+2 must now be evaluated as a candidate design rather than assumed to be the answer.

Capacity planning reduces exposure to cascading failure but cannot contain every overload. Major infrastructure loss, a network partition, a balancing fault or unexpected traffic can concentrate more work in one location than the plan anticipated. Admission and degradation controls must protect the service when those conditions arise.

Autoscaling can amplify a bad signal

Scaling on demand adds a control loop around capacity. Its metrics, startup delay, and dependency effects create the distinct failure modes below, which need different protections.

An autoscaler can fail to add enough capacity when its metric averages busy serving instances together with idle instances that are warming up or stuck. The SRE workbook describes this dilution failure mode; it suppresses scaling only when the resulting decision no longer calls for enough growth. Actual implementations differ. Kubernetes HPA’s CPU path sets aside certain not-yet-ready or missing observations and then recalculates conservatively, which can dampen scale-up rather than universally prevent it. Check metric population, readiness, initialization windows, and thresholds in the deployed scaler. A load-balancer capacity signal, suitable startup handling, and separately configured autohealing can help, but their behavior also needs verification.

Growth and shrinkage have different immediate risks. Failing to scale up can overload the service and discard useful traffic. Keep operating load below critical bottlenecks, allow enough time for new instances to become ready, and reserve capacity for overload protection and redundancy before relying on the scaling loop.

Scaling can amplify faults. The SRE workbook describes a release that consumes CPU without useful work: the autoscaler repeatedly enlarges the job until it wastes the available quota. In another scenario, a failed dependency leaves requests holding resources; adding frontends sends more traffic to that dependency and obstructs its recovery. Set minimum and maximum instance bounds with matching quota, assess the backends that will receive the added demand, and separate microservice quotas where a shared pool could be exhausted. Provide an obvious, fast, documented kill switch and train on-call engineers to scale manually.

These failures need different diagnoses. Measurement dilution can hide real demand; a CPU-consuming bug can create demand that more instances reproduce; and a downstream bottleneck can make more frontends harmful. Examine the signal and the resource being added instead of assuming that every scaling failure means additional capacity could never help. Control loops can oscillate when their delays and reactions interact, but oscillation is not inevitable.

Deciding what to drop, before you have to

When demand exceeds capacity, something is not served. The only question is whether you chose which. Google’s answer is to make importance a property of the request itself, with four values.

Criticality	Meaning
`CRITICAL_PLUS`	The highest-priority requests, whose failure causes serious harm visible to users
`CRITICAL`	The production-job default; failure affects users but less severely than CRITICAL_PLUS. Provision for expected traffic at this level and above
`SHEDDABLE_PLUS`	Partial unavailability is accepted; the batch-job default permits attempts again minutes or hours later
`SHEDDABLE`	Frequent partial unavailability and occasional complete unavailability are accepted

Google reports that four values covered almost all of its services; adding levels would also add operating work to every criticality-aware control. Outgoing calls inherit the incoming criticality by default, with overrides where the business meaning changes. Assign the label at a trusted entry point that understands the work. This is often an HTTP frontend, but can also be a batch coordinator or another business service. A person waiting is useful context, not a definition of importance or information available only at a browser-facing edge.

Request importance and response urgency are independent. Search-as-you-type suggestions may be highly expendable yet useful only within a short latency limit: refusing a late answer can be preferable to queueing it. Request criticality explains assignment without unnecessary priority levels.

Per-customer limits constrain an individual caller, but a ceiling is not a guaranteed reservation. Google’s example gives a service 10,000 CPUs and four large clients limits of 4,000, 4,000, 3,000, and 2,000 CPU seconds per second, with a stated 500 limit for each other user. The four named limits alone total 13,000. If they peak together, the service is short even though none exceeds its own quota. The source relies on peaks not coinciding; that assumption cannot guarantee that compliant customers remain unaffected. A design that promises protected service needs sufficient reserved capacity and an enforced allocation rule. Define how priority, fair sharing, and rejection apply when aggregate demand exceeds supply, including demand within individual limits. noisy neighbor covers this form of shared-resource contention.

Rejecting requests is not free

Refusing a request still uses resources. Protocol overhead can make rejection nearly as costly as a simple memory lookup, although rejection usually costs much less than full processing. A sufficiently large rejected population can exhaust a backend whose CPU is spent mostly saying no. Include that work when sizing overload protection.

Google’s adaptive throttling uses two minutes of client history: attempts made by the application and requests accepted by the backend. Once attempts exceed accepted requests multiplied by K, the probability of local rejection rises. Local rejection avoids network and backend work, while the attempts count still includes that demand. A lower K reduces backend rejection overhead more aggressively. The book’s preferred K of 2 allows extra probes, spending resources so clients can notice recovery sooner. This is a feedback tradeoff; a client sending very few requests has little evidence about backend state.

The book combines a per-request budget of three attempts with a per-client retry budget. Three attempts alone can bring offered traffic close to three times the original demand when most calls fail. The client-wide cap greatly reduces that amplification, but specify its denominator: retries limited to 10% of original calls allow at most 1.1 times as many attempts, whereas retries limited to 10% of all attempts allow about 1 / 0.9, or 1.111 times. The source’s 1.1 figure is a useful approximate illustration, not permission to ignore the extra load near saturation. Backends can also inspect attempt-count histograms and return a nonretryable overload response when widespread retries suggest that another task is unlikely to help.

A retry is useful only if another destination plausibly has capacity. When most backend tasks in a datacenter are overloaded, return the error to the caller rather than multiplying demand. If only a small subset is overloaded, an immediate attempt at another task can redistribute work. Space subsequent retries with exponential backoff and jitter to avoid synchronized arrivals. Throttling limits demand at the client; load shedding refuses it at the server.

Serving less instead of serving nothing

Graceful degradation preserves a reduced result by doing less work. The SRE book’s examples search a smaller candidate set, serve a cheaper local copy despite possible staleness, or use a faster ranking algorithm with lower accuracy. Decide which reductions remain useful and acceptable to the user before enabling them under load.

Emergency paths need operational experience. A degradation mode unused during normal traffic receives less exercise, so defects and unexpected behavior can remain hidden until overload. This risk applies to the other rarely used controls as well.

Regularly exercise degraded mode on a small, controlled server population near overload. Monitor how many instances enter it and alert when the extent exceeds the accepted service condition. Keep the mechanism understandable and provide quick disable and tuning controls, since complex protection can create its own faults. Graceful degradation develops the design decisions.

Design degraded mode around the metrics that trigger it, the work it changes, and the layer that enforces those changes. Decide whether each layer needs protection or a high-level admission point is sufficient; duplicated controls can interact just as the balancer and shedding policy did in Dressy.

The tools interact, and that is where outages come from

Load balancing, shedding and autoscaling all influence the distribution and stability of load. Separate installation and configuration do not make their behavior independent: each changes the demand or capacity that the others observe.

In the fictional Dressy example, the workbook fixes the balancer’s interpretation by assigning an overload error a virtual cost corresponding to 120% CPU utilization. This makes rejection signal overload instead of efficiency. That number belongs to this example’s balancing calculation; it is neither measured CPU consumption nor a requirement that every balancer price errors above successful requests. Other designs can use explicit admission capacity or overload signals. The required property is that shedding does not attract still more traffic by making a saturated destination look cheap.

Coordinate scaling thresholds, failover reserves, and propagated deadlines so each control protects the capacity that the others rely on.

Scale before you shed. Set thresholds to begin adding usable capacity before rejection starts, where the startup time permits it. Otherwise the service may discard work that timely growth could have handled.
Keep a floor in every location. Latency-based balancing can concentrate demand in one location and prompt that location to grow. If it fails, other locations cannot necessarily expand quickly enough to accept the displaced work. Maintain a minimum local instance count with tested spare failover capacity.
Propagate deadlines and enforce the operation contract. Stop obsolete reads cooperatively so they do not consume capacity needed by useful work. For accepted operations with effects, distinguish the response deadline from completion and recovery obligations; an RPC cancellation does not roll back a committed action.

When adding a load-control mechanism, examine its interactions with existing controls and instrument their shared boundaries. Monitor for feedback loops and coordinate emergency shutdown paths. Preparing those safeguards before deployment avoids having to discover their necessity during a later incident review.

What to do first

The SRE book gives overload defenses in a rough priority order, beginning with realistic load tests of both capacity limits and failure behavior. The test establishes which resource runs out first and how the resulting exhaustion affects requests; intuition alone cannot reliably establish those outcomes.

Use tests to establish the utilization knee, limiting resource, rejection cost and queueing delay for the deployed system. The remaining rough priorities are useful degraded results, early inexpensive rejection at the server, rejection at higher layers that prevents backend overload, and capacity planning.

The SRE book cautions that a fixed rate limit may ignore service health and leave capacity unused or fail to contain an existing cascade. This is a limitation of that control, not of every rate-limiting system. A controller can adjust admission using queue length, in-flight requests, resource pressure, or dependency health. Fixed customer quotas and adaptive overload protection answer different questions and can be used together; a QPS ceiling alone does not bound every resource.

The queue probability follows from the stationary M/M/1 state probabilities under the assumptions stated in that example. Apply those assumptions only where the workload supports them; the Dressy illustration and Google’s CPU, queue and retry figures likewise need their stated workload and implementation conditions.

A sequence a team can actually complete, assembled from the above rather than from ambition. Measure capacity in the resource that runs out first, and check that your load test’s request mix resembles production. Find the breaking point deliberately, and watch which resource goes and what the failure looks like. Size queues small, or go without them, unless your traffic is genuinely bursty. Set deadlines and propagate them. Put in the two retry budgets. Classify traffic into a handful of criticality levels, set them at the edge, and let them propagate. Then build one degraded mode, and arrange to exercise it on purpose.

And a caution about the middle of that list, because overload protection is itself a source of outages. Every mechanism here changes what your metrics mean: shedding flattens CPU, throttling hides demand at the client, degraded mode makes latency look better while serving worse answers. Adding a protection without adding the observation that makes it visible converts a loud failure into a quiet one. When a service is at 40% CPU and users are unhappy, the first thing to check is which of your safety mechanisms is engaged.

Dressy had spare capacity in B and C, but its balancer interpreted refusal as efficient processing and concentrated traffic in A. The case shows why load management must observe offered demand, accepted work, rejected work, and remaining capacity together. Keep the workload and acceptance conditions attached to a QPS capacity figure, and pair it with the limiting resource signals. Decide which work can be delayed, degraded, or refused before the system reaches its limit.

References

Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.