Cascading Failure

A cascading failure is one that grows because of the system’s own reaction to it: a component becomes overloaded, its failure increases load on others, and their failures increase load further. The mechanism is a feedback loop rather than a sequence of independent breakages.

Google’s Site Reliability Engineering devotes a chapter to the pattern, and its distinguishing property is worth memorizing because it shapes how incidents are handled. A cascading failure can persist after the original trigger has been removed. Restoring the thing that broke first does not restore service, because the load that is now keeping the system down is being generated by the system itself.

The common amplifiers are ordinary reliability measures behaving badly at scale. Retries multiply traffic exactly when capacity has shrunk. Health checks fail under load, so instances are removed, so the remaining instances carry more. Failover shifts an already-excessive load onto a replica, which then fails the same way.

Because the loop is the problem, the remedies are ones that reduce demand rather than add capacity. Load shedding rejects work cheaply so the accepted work can complete, and retry budgets and attempt limits cap how much traffic amplification is possible at all. Backoff has to be exponential and jittered so that retries spread out instead of arriving together; a cap on backoff does a different job — it keeps waits from growing without bound, and clients sitting at that cap go on retrying at a fixed interval, so the cap is not what protects a recovering service.

Recovery often requires deliberately reducing traffic before restoring it — draining queues, disabling retries, or bringing the system up while most requests are still being rejected. That is counterintuitive during an incident, which is the reason it belongs in a runbook written beforehand rather than in a judgment made at three in the morning.

How this interacts with timeouts and retry design is worked through in Partial Failure, Timeouts, and Retries.