Retry Budget

A retry budget limits retries as a fraction of overall traffic rather than as a fixed number per request. It keeps retries useful against scattered transient failures while preventing them from multiplying load when the dependency is in real trouble.

The problem

A per-request limit such as “make up to 3 attempts, including the first call” looks safe, and it is for a single request. Across many requests it is not. During an outage almost every request fails, every request uses all of its attempts, and traffic to the failing service becomes three times normal at the moment it can least cope. When layers retry independently, the factors multiply: three layers with three attempts each can send up to \(3^3 = 27\) calls for one user action. This self-inflicted overload is a retry storm.

The idea

Cap the share of calls that may be retries. Google’s SRE book describes two limits used together: a per-request limit (for example, at most three attempts) and a per-client limit, where a client retries only while retries stay below about 10% of its requests. When most requests succeed, there are few retries and the budget is rarely binding. When failures are widespread, the budget runs out and further failures go to the caller instead of the dependency. The book notes that the 10% ceiling reduces the worst-case amplification from about 3x to about 1.1x.

Implementation

A common version is a token counter. Every original request adds a fraction \(\rho\) of a token (here \(\rho = 0.1\)), up to a cap. Each retry consumes one token and may proceed only if a token is available. In steady state, retries cannot exceed \(\rho\) times the number of requests, plus the stored tokens. The first attempt is never limited by the budget.

Simulation

10,000 requests, up to 3 attempts each, with an independent probability \(p\) that each attempt fails, for a budget with \(\rho = 0.1\) and a cap of 10 tokens:

With rare failures, the budget changes nothing, since 5% failures need about 5% retries, well within the allowance. With a total outage, the per-request policy sends three times the traffic and gets nothing for it, whereas the budget holds the increase to about 10%. The middle row shows the cost: when half of the calls fail, retries would have rescued many requests, and the budget refuses some of them, so the success rate is lower (54.9% versus 87.4%). The budget deliberately gives up some availability to avoid pushing a struggling service over. If the failures are independent, the dependency can handle the extra load, and a larger \(\rho\) is appropriate; during correlated failures, a small \(\rho\) is protective.

Placement and tuning

• Budget per client process, per destination service, so that one failing dependency does not use the budget for a healthy one. Gateways and proxies can enforce it per upstream cluster.
• Apply it at one layer where possible, or let the lower layers make no retries. A budget at each of three layers still limits each layer’s amplification to 1.1x but compounds to about 1.33x overall.
• Combine it with backoff and jitter, which spread retries in time, and with a circuit breaker, which stops calls entirely under sustained failure. See Circuit Breaker and Deadline Propagation.
• Retries are for transient errors on idempotent operations; do not spend budget on errors that cannot clear by retrying, such as 400 responses or exhausted quotas.

Neighbors

The per-request retry count limits the work of a single request; the budget limits the work of the whole client. Rate limiting, such as Token Bucket, restricts total request volume, and a retry budget is a specialized token bucket for retries only. Load shedding is the server-side counterpart: the server drops excess work, and the retry budget keeps clients from sending it.

References

• Google SRE Book, Handling Overload