Circuit Breaker

A circuit breaker is a wrapper around calls to a dependency that stops making the calls when the dependency appears to be failing. Like its electrical namesake, it trips under fault conditions so that the damage does not spread, and it resets once the fault is gone.

Why it exists

When a dependency is down or very slow, every caller that keeps trying pays twice: it waits for a failure that will almost certainly happen, and it adds load to a service that is trying to recover. Threads, connections and request budgets fill up with doomed calls, and the caller’s own users see the latency. A breaker converts “wait for the timeout, then fail” into “fail immediately”, which frees the caller’s resources and gives the dependency room to recover.

Three states

• Closed. Calls pass through. The breaker counts failures; a success resets the count (in the simple version, which counts consecutive failures).
• Open. After the failure count reaches a threshold, the breaker trips. Calls are rejected at once, without touching the dependency, typically with a fallback result or an error that the caller can handle.
• Half-open. After a cooldown period in the open state, the breaker lets a limited number of trial calls through. If they succeed, it returns to closed. If one fails, it goes back to open and the cooldown restarts.

Fowler’s description of the pattern uses these states, with a failure count threshold and a reset timeout.

Trace (simulated)

A dependency is healthy until t = 5 s, fails from 5 s to 30 s, and recovers afterward. The client sends one request per second for 60 seconds. Breaker settings: open after 3 consecutive failures, cooldown 10 s.

t=5,6,7   calls fail                 -> closed -> open at t=7
t=8..16   rejected locally           (9 requests, no calls to the dependency)
t=17      probe fails                -> open (cooldown restarts)
t=18..26  rejected locally
t=27      probe fails                -> open
t=28..36  rejected locally
t=37      probe succeeds             -> closed

Over the 60 requests, 33 reached the dependency (5 healthy calls before the outage, 3 failures that tripped the breaker, 2 failed probes, and 23 successes from the successful probe at t = 37 onward) and 27 were rejected locally. Without a breaker, all 60 would have reached it, and the 25 requests made during the outage (t = 5 to 29) would each have waited for a failure. The breaker also keeps the outage visible longer than it lasts: the dependency recovered at t = 30 and the breaker only closed at t = 37, because probes happen only at cooldown boundaries. That delay is the price of the protection, and a shorter cooldown reduces it at the cost of more probe failures.

Choosing thresholds

• A consecutive-failure count is simple but trips on short bursts at low traffic and may never trip at high traffic with scattered errors. A failure rate over a sliding window (for example, 50% of the last 20 calls, with a minimum number of calls) behaves more predictably as traffic varies.
• Count only failures that indicate a sick dependency: timeouts, connection errors and 5xx responses. A 400 caused by a bad request says nothing about the dependency’s health, and counting it lets one malformed client trip the breaker for everyone.
• The cooldown trades recovery time against probe load. Limit the number of concurrent half-open probes so that a recovering dependency is not hit by a full wave of traffic at once.
• Scope the breaker narrowly: per dependency, and where the dependency is partitioned, per endpoint or per tenant, so that one failing partition does not disable the healthy ones.

Interaction with retries and fallbacks

A breaker sits outside the retry loop, or every retry of every request counts as a separate call. Retries absorb isolated transient failures; the breaker handles sustained ones. When the breaker is open, the retry loop should treat a rejection as non-retryable, since retrying the breaker only burns the deadline. See Retry Budget for limiting retry volume. A rejected call needs somewhere to go: a cached answer, a simpler rule-based path, a queue for later, or an explicit error. Together the breaker and the fallback define the degraded mode of the system.

Limits

A breaker reacts after failures have happened, so the first few calls still fail. It does not make a dependency healthy; it limits the harm. It also adds state that has to be monitored: breaker transitions are useful alerts, because an open breaker means users are on the fallback path.

References

• Martin Fowler, Circuit Breaker