Cascading Failure

A cascading failure is one that grows because of the system’s own reaction to it: a component becomes overloaded, its failure increases load on others, and their failures increase load further. The mechanism is a feedback loop rather than a sequence of independent breakages.

Google’s Site Reliability Engineering devotes a chapter to the pattern, and its distinguishing property is worth memorizing because it shapes how incidents are handled. A cascading failure can persist after the original trigger has been removed. Restoring the thing that broke first may not restore service, because the load that is keeping the system down can be generated by the system itself.

The common amplifiers are ordinary reliability measures behaving badly at scale. Retries multiply traffic exactly when capacity has shrunk. Health checks fail under load, so instances are removed, so the remaining instances carry more. Failover shifts an already-excessive load onto a replica, which then fails the same way.

Because the loop is the problem, reducing demand is a central recovery measure. Load shedding rejects work cheaply so the accepted work can complete, and retry budgets and attempt limits cap how much traffic amplification is possible at all. A common retry policy uses exponential backoff with jitter so that retries spread out instead of arriving together; a cap on backoff does a different job — it keeps waits from growing without bound. Without jitter, clients at the cap can retry at a fixed interval. With full jitter, each wait is sampled again between zero and the capped upper bound, so reaching the cap does not make the intervals fixed. Neither variant limits the total number of attempts: retry budgets and attempt limits address that separate risk.

Adding capacity is not excluded, and the chapter’s own worked example ends with it: when a broadcast pushed traffic past what one datacenter could serve, the responders restored the service by raising the number of serving tasks there. What decides whether that works is where the capacity lands. It helps when the exhausted resource is the one being added to, when the addition is available in minutes rather than after a scaling delay, and when the new instances do not have to warm a cache or open a connection pool through the same saturated dependency before they can serve. It makes things worse when the bottleneck is shared — more frontends in front of one overloaded database simply send the loop more traffic — and when the new instances join a pool whose health checks are already failing under load. In practice the two move together: shed first so the system stops amplifying, add capacity to the real bottleneck, then let load back in.

Recovery often requires deliberately reducing traffic before restoring it — draining queues, disabling retries, or bringing the system up while most requests are still being rejected. That is counterintuitive during an incident, which is the reason it belongs in a runbook written beforehand rather than in a judgment made at three in the morning.

How this interacts with timeouts and retry design is worked through in Partial Failure, Timeouts, and Retries.

References

• Google, Site Reliability Engineering, Addressing Cascading Failures
• AWS Architecture Blog, Exponential Backoff and Jitter