Capacity Planning

Capacity planning is deciding how much of a resource to provision, where, and by when. Google’s SRE book describes the conventional version as a four-step cycle — “collect demand forecasts,” “devise build and allocation plans,” “review and sign off on plan,” and “deploy and configure resources” — typically looking “anywhere from several quarters to years” ahead.

The first thing worth internalizing is that the plan does not converge: “it bears stressing that capacity planning is a neverending cycle: assumptions change, deployments slip, and budgets are cut, resulting in revision upon revision of The Plan.” And revisions compound — “a shortfall this quarter must be made up in future quarters.” A team that treats a changed plan as evidence of poor planning has misunderstood the activity.

The arithmetic, and where the numbers come from

The part that is genuinely simple is the redundancy calculation, and the documented example is worth working through because most teams state a target like N+2 without converting it into machines: “if every cluster’s breaking point is 5,000 QPS, the load is evenly spread across clusters,” and “the service’s peak load is 19,000 QPS, then approximately six clusters are needed to run the service at N + 2.”

Rounding 19,000 / 5,000 up gives four clusters; adding two gives six. This expresses the intended ability to carry peak after losing two clusters, not a requirement that two sit idle. In an active-active arrangement, six clusters share about 3,167 QPS each, rising to 4,750 on each of four survivors. The calculation assumes that surviving capacity and routing can actually absorb the redistributed load.

The breaking point reveals a failure boundary, not necessarily the highest load that meets latency and error objectives. Measure both under a representative workload. If the acceptable operating capacity is 4,000 QPS rather than the breaking point of 5,000, the same simplified N+2 calculation gives ceil(19,000 / 4,000) + 2 = seven clusters. Validate failure domains, startup and cache warm-up, dependencies, and traffic redistribution as well as the count. The SRE book recommends realistic tests of both capacity limits and overload behavior.

QPS is meaningful when the workload and acceptance criteria are stated. Google’s overload chapter warns that an aggregate request count can become a poor proxy for resource consumption as request costs and their mixture change. Track CPU, memory, I/O, and other limiting resources alongside throughput. Google found CPU useful for many of its services; another service may reach a different constraint first.

Three forecasting errors follow from those conditions: retaining a QPS estimate without its workload assumptions; projecting total growth while missing a more expensive request mix; and deriving peak demand from an average when the peak-to-average relationship is changing. A new mix or release calls for revalidation rather than proving the old capacity estimate wrong in advance.

Why the conventional process hurts

The SRE book names two weaknesses, and both are recognizable from the inside.

Brittle plans. Google describes how changes in service efficiency, adoption, infrastructure delivery, or performance targets can invalidate allocations. Small changes may require checking the whole plan; larger changes may require rebuilding it. The important dependency is that one period’s supply shortfall changes what later periods must deliver.

Laborious allocation. Forecast collection can be slow and error-prone, while location and latency requirements prevent resources from being freely interchangeable. A surplus in North America may not serve demand that must stay in Asia. Constrained allocation also includes bin packing, which is NP-hard in general. The book describes the additional difficulties of maintaining these plans in spreadsheets: stale data, limited checking, and difficult change tracking.

Google reports that teams sometimes simplified requirements merely to make the planning process tractable. That is a reason to preserve the service constraints explicitly when improving planning tools; an easier allocation process is not an improvement if it silently drops the requirement it was meant to satisfy.

State the requirement, not the resource

Google’s intent-based approach records why capacity is needed and which constraints admit flexibility, then uses those requirements to generate allocations. This makes changes in demand, supply, or service goals inputs to replanning instead of instructions for manually editing every resource request.

Even without any tooling, the ladder of abstraction they give is a useful exercise, because each rung exposes what the one below concealed.

1. “I want 50 cores in clusters X, Y, and Z for service Foo.” An explicit resource request — and, as the source asks, “why do we need this many resources specifically in these particular clusters?”
2. “I want a 50-core footprint in any 3 clusters in geographic region YYY.” Easier to fulfill, “although it doesn’t explain the reasoning behind its requirements.”
3. “I want to meet service Foo’s demand in each geographic region, and have N + 2 redundancy.” Now “we can understand at a more ‘human’ level what happens if service Foo does not receive these resources.”
4. “I want to run service Foo at 5 nines of reliability.” The most abstract, and the one where “the ramification if the requirement isn’t met becomes clear: reliability will suffer” — and where N+2 might turn out not to be the right shape at all.

The higher-level statements expose what can be negotiated: placement, redundancy, cost, and the service outcome. The book describes automation reducing manual resource allocation so service owners can focus on SLOs, dependencies, and infrastructure needs. Even without such tooling, stating the rationale makes an allocation request easier to review.

Plan for both overload protection and the specified infrastructure losses. Record whether those scenarios must be survived together and avoid silently counting the same headroom twice. For autoscaling, include minimum and maximum sizes, available quota, and time to become useful; the SRE workbook recommends these constraints alongside spare capacity.

What it does not buy

This is the sentence to keep, because it ends the argument that overload is a procurement problem:

Capacity planning reduces the probability of triggering a cascading failure, but it is not sufficient to protect the service from cascading failures. When you lose major parts of your infrastructure during a planned or unplanned event, no amount of capacity planning may be sufficient to prevent cascading failures. Load balancing problems, network partitions, or unexpected traffic increases can create pockets of high load beyond what was planned.

Capacity planning can include explicit failure scenarios as well as a demand forecast. The N+2 example above plans for two cluster losses; such a loss is still an event to detect and manage even when surviving capacity is adequate. Residual risk includes failures outside the chosen envelope, inaccurate capacity assumptions, and routing or overload controls that do not behave as tested. Verify the actual redistribution and service results during an incident. Planning and runtime protection address different parts of the problem; neither the planned resource count nor the fact that a failure was anticipated guarantees a successful response.

Autoscaling does not change this either: “some systems can grow the number of tasks for your service on demand, which may prevent overload; however, proper capacity planning is still needed” — because scaling has bounds, quota, warm-up time, and downstream dependencies that were provisioned by someone else.

A small organization can begin by measuring acceptable operating capacity and the later failure boundary under a stated workload. Translate the required failure tolerance into capacity that remains usable after those losses. Track demand in request and resource terms, including peak composition. State how redundancy and overload margins cover separate or simultaneous scenarios, and revisit the assumptions when workload, software, or infrastructure changes.

Review the measurement unit, workload, service acceptance threshold, covered failures, and response to demand beyond the forecast. The capacity count alone does not choose what to refuse: request criticality, graceful degradation, and load shedding develop those responses. The Shedding Region Looked Healthiest shows how the controls interact.

References

• “Software Engineering in SRE,” Google SRE Book
• Google SRE Book — “Addressing Cascading Failures”
• Google SRE Book — “Handling Overload”
• “Managing Load,” Google SRE Workbook

The QPS and cluster figures are the sources’ worked examples.