Partial Failure: What Breaks When Systems Talk Over a Network

A checkout service calls a payment provider to authorize a card. The call takes longer than the three seconds the client was willing to wait, so the client gives up and returns an error to the customer. What happened to the authorization? The caller cannot tell from the timeout alone. The request may never have arrived. It may have arrived, been authorized, and the response lost on the way back. It may still be running. The customer, seeing an error, presses the button again.

This is partial failure: in a system whose parts talk over a network, one part can fail or become unreachable while the rest keeps running, and the surviving parts cannot tell which of several situations they are in. A call within one process can also fail in complicated ways — it can throw, hang, or leave half its work done — but the caller at least observes the same event the callee experienced. A network call splits into three things that can fail separately: delivering the request, executing it remotely, and delivering the response. Because the caller only sees the last of these, its knowledge of the outcome and the outcome itself come apart, and most of the machinery in distributed systems exists to manage that gap.

This article works through what a timeout actually tells you, why the obvious response of retrying makes overload worse, and which defenses contain the damage. The examples are invented for illustration. Replication, consistency, and the ordering of events raise a second set of problems that deserve their own treatment.

A timeout is an absence of information

Teams often treat a timeout as a kind of error, alongside “invalid input” or “not found”. It is a signal, but not the same kind. Those errors are answers about the operation: the callee considered the request and reports what it decided. A timeout reports only that the caller’s own waiting limit was reached. It says nothing about the remote work, which may not have started, may have finished, or may still be running.

What may have happened	State of the work	What a retry does
The request never arrived	Not started	Performs the work once, if the retry itself succeeds
The request arrived and failed before any side effect, or was fully rolled back	Not done, nothing left behind	Performs the work once, if the retry itself succeeds
The request arrived and failed partway, after some effects were already applied	Partially done	Repeats the effects already applied unless the operation is idempotent; a partial state may also make the retry fail differently
The request succeeded, the response was lost	Done	Performs the work twice unless the operation is idempotent
The request is still being processed	In progress, possibly with effects already applied	Two concurrent executions, which may conflict with each other

Two of these rows are usually collapsed into one, and that is where implementations go wrong. “It failed” is not the same as “nothing happened.” An operation that writes a row, calls a payment provider, and then fails has left both effects behind. Whether the second row or the third applies depends on whether the operation is wrapped in something that rolls back — a database transaction covers its own writes, but it does not undo a message that was published or a charge that was made.

The retry column is also worth reading carefully: a retry changes the risk of duplication, but it does not promise success. If the cause was an overloaded dependency, the retry may well time out too. Duplication risk and probability of success are separate questions, and a design has to answer both.

The caller cannot distinguish these cases from the outside, and no timeout setting removes the ambiguity. A longer timeout gives the cases that are still unfolding — the request still being processed, or a response merely delayed rather than lost — more chance to resolve before the caller gives up. It does nothing for a response that is genuinely gone. It also holds the caller’s resources longer, which matters when the callee is slow because it is overloaded. A shorter timeout frees the caller sooner and increases the number of requests that are abandoned while still running.

Two practical rules follow. First, a timeout should be derived from the target it protects, not chosen by habit: if the caller’s own response latency target is two seconds, a downstream call cannot be given five. Second, whatever the caller does next has to be safe in every case in that table, because it does not know which one it is in.

Why retries make things worse before they make them better

Retrying is the natural response to an uncertain outcome, and for a transient fault it works. The problem is what happens when the cause is not transient. The SRE book names server overload as the most common cause of cascading failures, and an overloaded dependency is exactly the situation in which retries add load at the moment the system has none to spare.

The Google SRE book defines a cascading failure as one that grows over time through positive feedback: a portion of the system fails, which raises the chance that other portions fail. Its example is a set of replicas behind one service, where the failure of one replica pushes its share of traffic onto the others, making them more likely to fail in turn, until the whole set is down. Retries are one of the strongest feedback loops available. If every client retries twice on failure, a service that starts dropping requests immediately receives up to three times its normal traffic.

The same chapter describes how the pressure travels through a server’s resources. When requests arrive faster than they can be served, queues grow, so latency rises and memory use with it. Threads wait on locks or on slow backends, and thread starvation can make health checks fail, which takes the instance out of rotation and sends its traffic to instances that are already struggling. In garbage-collected runtimes the book describes a further loop it calls the GC death spiral: less available CPU makes requests slower, which raises memory use, which causes more collection, which leaves even less CPU.

The three-times figure above assumes retries happen in one place. They rarely do. The same chapter warns that retries issued at several levels of a stack multiply rather than add: a single request at the top can produce as many attempts as the product of the attempts at each layer. Its example is a database behind a backend, a frontend, and JavaScript in the browser, each configured for three retries — four attempts each — so one user action can reach the database 4³ times, which is 64 attempts. Nobody designs that. It is assembled by three teams who each added a sensible retry.

Two practices keep this contained, and both are decisions about the system rather than about any one client. The first is to decide which layer owns retrying and have the others pass failures through, which the book frames as thinking about the service holistically and asking whether retries at a given level are really needed. The second is deadline propagation: rather than each server inventing a fresh timeout for its own backend calls, a deadline set at the top of the stack travels with the request, and each level works within what remains of it. Without that, a client can give up while three layers below it are still retrying against a deadline of their own.

Retries interact badly with timeouts in one more way. A client that retries after a timeout without cancelling the original request leaves the first execution running. The server is now doing work that no one is waiting for, which is the worst possible use of a saturated system. Deadline propagation helps here too, because a request whose deadline has passed can be abandoned rather than completed for nobody.

Making retries safe: backoff, jitter, and budgets

Three well-documented mechanisms turn retries from an amplifier into a useful tool.

Capped exponential backoff multiplies the wait after each attempt up to a maximum. It reduces the rate at which any one client retries, but on its own it leaves clients synchronized. Marc Brooker’s 2015 analysis on the AWS Architecture Blog simulates many clients competing for the same resource and shows that with backoff alone the calls still arrive in clusters, because every client is waiting the same intervals. The fix is jitter: randomizing the wait. In his simulation, adding jitter cut the number of calls by more than half and also improved the time for all clients to finish, which is why the recommendation is backoff and jitter together rather than backoff alone. He compares several forms, including full jitter, where the wait is a random value between zero and the current cap, and decorrelated jitter, which grows the random range based on the previous wait.

Retry budgets limit retries across the system rather than per call. The SRE book describes two used at Google. A per-request budget allows up to three attempts, on the reasoning that a request which has already landed on three overloaded tasks is unlikely to be helped by a fourth. A per-client budget tracks the share of a client’s outgoing requests that are retries and stops retrying when that share exceeds 10 percent, on the reasoning that widespread retrying means the problem is not a few unlucky requests. The specific numbers are Google’s; the principle is that a client should notice when retrying has stopped being an exception.

The SRE book also recommends a server-wide retry budget, giving the example of allowing only 60 retries per minute in a process, so that retry amplification cannot destabilize the system even when individual clients behave badly.

Finally, retries should be reserved for errors that a second attempt could plausibly resolve, and that distinction is about the cause rather than about whether the response was an error. An invalid field or a failed authorization will fail identically on every attempt until the request itself changes, so retrying it only spends capacity. A rejection caused by rate limiting is the opposite case: RFC 6585 defines HTTP 429 as the client having sent too many requests in a given amount of time, and says the response may include a Retry-After header telling the client how long to wait before making a new request. That is an explicit invitation to try again later. When a service provides such a hint, clients should honor it rather than apply a schedule of their own.

Idempotency: making duplicate work harmless

Backoff and budgets control how often a retry happens. They do nothing about the cases where the work already ran — the one where it succeeded and the response was lost, and the one where it failed after applying some effects. These cases are handled by making the operation idempotent, meaning that performing it again produces the same result as performing it once.

Some operations are naturally idempotent. Setting a customer’s status to “closed” can be repeated without harm. Adding 50 dollars to a balance cannot. For operations that are not naturally safe, the standard device is a client-generated key that lets the server recognize a repeat. Stripe’s API documentation describes the mechanics clearly: the client generates an idempotency key, the server stores the status code and response body of the first request made with that key, and later requests carrying the same key receive the same stored result, including a stored 500 error rather than a fresh attempt. Stripe suggests random values such as version 4 UUIDs, prunes keys after at least 24 hours, and returns an error if the same key arrives with different parameters, which catches a client accidentally reusing a key for a different operation.

Three details decide whether this works in practice.

• Where the key comes from. The caller must generate the key before its first attempt and reuse it for every retry of that attempt. A key generated per attempt provides no protection.
• How long the record is kept. Deduplication only covers the window in which keys are retained. A retry that arrives after the window looks like new work. Stripe’s 24-hour window is a product decision, and a system that replays days-old messages needs a different approach.
• What happens to concurrent duplicates. Two copies of the same request can arrive while the first is still running. Rejecting or blocking the second is safer than allowing both to proceed and hoping the storage layer sorts it out.

In data pipelines the same idea appears under different names, and the choice of key is where it usually goes wrong. A consumer that may see a message twice performs event deduplication, but deduplicating on a business key alone throws away real events: an order legitimately produces a creation, an amendment, a cancellation, and a refund, and all four share the order ID. What identifies a duplicate is the same logical event arriving twice, so the key has to be either an event identifier assigned once at the source and carried through every redelivery, or the business key combined with something that distinguishes one event from the next, such as a version, a sequence number, or the source log position.

Batch loads need the equivalent. A run has to be identifiable by what it covers — the target partition or time window — and a rerun of that window has to replace or merge its rows rather than append them, whether by overwriting the partition or by keying an upsert. That is a separate guarantee from atomic publication, which ensures readers see either the old data or the new and never a half-written mixture. Atomic publication says nothing about whether the new data contains yesterday’s rows twice. A pipeline needs both: an identified, replaceable unit of work, and a visible switch between versions of it.

Shedding load and pushing back

A service under more load than it can handle has only bad options, so the useful question is which bad option it chooses deliberately. The SRE book’s advice is to fail early and cheaply when overloaded, to serve degraded but cheaper results where the product allows it, and to reject work at higher levels rather than letting servers collapse. It also notes that rejecting is not free: for a cheap request, refusing it can cost nearly as much as serving it, which limits how much a server can protect itself by rejection alone.

Rejecting work is one direction of a broader idea. Backpressure is the general mechanism by which a slow consumer tells a fast producer to slow down, rather than accumulating an unbounded queue between them. Where the queue lives determines what happens when it fills.

Where work waits	What happens when the consumer is too slow	What to decide in advance
In memory inside the consumer	Latency and memory grow, then the process fails and loses the queued work	A bounded queue size and what to do when the bound is reached
In a broker or durable queue	Backlog grows and consumers fall behind. Work already acknowledged as stored survives, subject to the broker’s durability settings and available capacity, until retention expires or the volume exceeds what the broker can hold	Retention long enough to survive a realistic outage, the acknowledgement and replication settings that make a write durable, headroom for the backlog, and lag alerting
At the producer	The producer blocks or buffers, which can push the problem upstream to its own callers	Whether the producer may drop, block, or store locally, as with a persistent sending queue
Rejected at the edge	Callers receive an explicit error quickly and capacity is preserved for the rest	Which requests are shed first, and how clients are told to back off

Every system makes this choice, whether or not anyone decided it. A system with no explicit policy has chosen the first row by default.

What these defenses cost

Each mechanism above is usually presented as a good practice, which hides the fact that every one of them adds something to build, operate, and get wrong. A design review that adds all of them without noticing the bill tends to produce a system that is harder to run than the failures it was protecting against.

Defense	What it costs	How it fails
Idempotency keys	A store that must be written on the request path, kept consistent with the operation itself, and retained for the whole deduplication window	The store becomes a new dependency on every write; if it is unavailable, either writes stop or deduplication silently stops
Backoff and jitter	Slower recovery for the caller, and latency that is harder to reason about because waits are randomized	Caps set too high turn a brief fault into a long outage for the user
Retry budgets	Configuration that has to be revisited as traffic changes	Set too low, genuine transient faults surface as user errors; set too high, the budget does not bound anything
Backpressure	The pressure moves upstream rather than disappearing, so the producer now needs a policy too	A chain of producers each blocking on the next, which turns one slow component into a stalled pipeline
Load shedding	Deliberate errors for some users, plus the work of deciding which requests matter less	Rejection is not free: the SRE book notes that refusing a cheap request can cost nearly as much as serving it, so shedding alone cannot save a saturated server
Durable queues	Storage, retention management, capacity headroom, and lag monitoring	Work the broker acknowledged as stored is delayed rather than lost, subject to its durability settings. Two separate limits end that: retention expiry, which discards silently once the window passes, and the capacity limit, whose behavior is configured — dropping the oldest messages, or refusing new publishes so the producer finds out. Know which one your broker is set to

The idempotency store deserves the most attention because it is the one that turns a safety mechanism into a dependency. Stripe’s design stores the status code and response body of the first request, which means the record has to survive at least as long as any client might retry, and has to be written in a way that cannot disagree with whether the operation actually happened. A key written before the work and never cleaned up after a crash blocks a legitimate retry. A key written after the work leaves a window in which a retry duplicates it. This is why the deduplication record and the operation are usually written in the same transaction where the storage allows it.

Where one transaction cannot span both, there is no single substitute, and it helps to separate three different jobs that get lumped together.

• Prevent the duplicate. Push the deduplication to the side that owns the effect: send an idempotency key the receiver honors, as Stripe’s API does, so a repeated request returns the stored result instead of charging again. This is the only one of the three that stops a duplicate charge or a duplicate notification from happening.
• Resolve the uncertainty. Before retrying, ask what actually happened: query the operation’s status by the caller’s own reference, or keep a durable record of in-flight work with its state so that a restarted process can resume rather than restart. This turns an unknown outcome into a known one.
• Detect and correct afterwards. Reconciliation compares what was sent against what was recorded and surfaces the mismatch. It is a safety net, not a defense: it finds the double charge, it does not prevent it, and reversing the effect is separate work that has to be designed too.

A path that crosses a transaction boundary usually needs the first two and keeps the third as a check. Where the intent must be recorded locally and delivered separately, the transactional outbox described later is the standard arrangement, and it works precisely because the receiver is idempotent.

None of this argues against the defenses. It argues for choosing them per crossing point rather than applying all of them everywhere. A call that is naturally idempotent, such as setting a value, needs no key store. A backlog that is acceptable for an hour needs lag alerting more than it needs backpressure.

Getting out once it has started

The defenses are preventive. When they are absent or insufficient and a service is already in a feedback loop, the problem changes character, and the most important thing to know is that such a system often will not recover on its own. The SRE book makes this point directly: if the condition that started the cascading failure has not been fixed, the failure will return shortly after traffic does. A system without enough global capacity re-enters the loop as soon as normal load comes back, which is why “we restarted it and it fell over again” is such a common incident note.

The chapter lists the immediate moves available, roughly in order of how blunt they are.

• Add resources if idle capacity exists, which buys time rather than fixing anything.
• Stop health-check-induced deaths. The book separates two kinds of check, and the distinction matters here. A process health check asks whether the process should be killed and restarted; under overload it can fail because the server is busy, so the orchestrator restarts instances that were still doing useful work, which costs their in-flight requests and their warm caches. Temporarily relaxing that check breaks the loop. A serving health check asks whether an instance should receive traffic, and turning that one off is not the same move: it sends requests to instances that genuinely cannot serve them. Adjust the check that is causing unnecessary restarts, not the one that is routing away from dead capacity.
• Restart wedged servers when they are stuck in a garbage collection spiral, a deadlock, or requests with no deadline. The book’s caution here matters more than the remedy: identify the source first, and make sure a restart will not simply move the load somewhere else. Roll it out slowly and watch.
• Drop traffic, which the book calls a big hammer. It is the bluntest and most dependable of these, and it requires someone with the authority to decide whose requests are refused. Bringing load back gradually afterwards matters as much as cutting it, because returning to full traffic at once re-enters the loop.
• Enter degraded mode and serve lower-quality results, which only exists as an option if someone built it before the incident.
• Eliminate batch and bad traffic. Turn off non-critical background work, and block the specific queries or clients causing disproportionate load.

Two of these are design decisions disguised as incident actions. Degraded mode and the ability to shed selectively have to exist beforehand. The practical consequence is that the question “what do we turn off first?” belongs in the design review, not in the incident channel, and the same chapter recommends finding the answer by load testing components until they break, testing both gradual increases and sudden spikes, and observing how a component returns to normal after being pushed past its limit.

Where partial failure shows up in data and platform work

The mechanics above appear in three settings that look unrelated until the shared cause is visible.

• Pipelines. A job that writes half its output and then fails leaves the destination in a state that is neither old nor new. The defenses are the same ones described above, applied to batches instead of requests: write to a staging area and publish atomically, record a checkpoint so a restart resumes rather than repeats, make each run idempotent so that a rerun of the same window is safe, and reconcile totals against the source to detect what slipped through.
• Service integration. A workflow that updates a local database and then calls another service has two outcomes to keep consistent with no shared transaction. A common remedy is to write the intent to a table in the same transaction and let a separate process deliver it, which is the transactional outbox. The delivery is at-least-once, so the receiver must be idempotent. Long-running work that was interrupted also needs a way to prevent a revived old worker from writing after a new one took over, which is what a fencing token provides.
• Operations. Alerting on “the pipeline failed” misses the more dangerous case, in which the job reported success while a downstream write was lost. Operational targets therefore need to cover completeness and freshness of the result, not only the exit status of the job, which is one reason service level objectives are written against user-visible outcomes.

A design review question that exposes most of these problems at once: for each call that crosses a process boundary, what happens if the caller times out, and what happens if the same request arrives twice?

A short checklist for a crossing point

Question	If the answer is unclear
What timeout does the caller apply, and how was it derived?	The caller’s own latency target cannot be met when the dependency is slow
Which errors are retried, how many times, with what backoff and jitter?	Retry storms during the next dependency slowdown
Is the operation idempotent, and who generates the key?	Duplicate charges, duplicate rows, duplicate notifications
What does the callee do when it is beyond capacity?	Queues grow until the process fails, losing in-flight work
Where does work wait, and for how long may it wait?	Unbounded backlog, or silent loss when retention expires
How is a failed or partial run detected and corrected?	Errors are discovered by a user reading a wrong number

Questions to explore further

• For your most important write path, what is the current behavior when the same request arrives twice, and how do you know?
• Which of your clients retry without jitter or without a budget, and what would their combined load look like during a five-minute dependency outage?
• Where does work queue up in your pipelines, and how long can it queue before something is lost rather than delayed?
• If your most important service had to shed half its traffic right now, which half would go, and who decides?
• Which of your write paths depends on a deduplication store, and what happens to those writes when that store is unavailable?

References

All sources were checked on September 15, 2026.

• Google, Site Reliability Engineering: Addressing Cascading Failures
• Google, Site Reliability Engineering: Handling Overload
• Marc Brooker, Exponential Backoff And Jitter (AWS Architecture Blog, 2015)
• Stripe, API reference: Idempotent requests
• RFC 6585, Additional HTTP Status Codes, Section 4: 429 Too Many Requests (2012)