Partial Failure

Partial failure is the condition in which some parts of a distributed system have failed while others continue working. It is the characteristic difference between a distributed system and a single program — though the contrast is one of degree rather than a clean line, and the distinction is worth stating precisely because it decides what you have to design for.

A single program crashing does not guarantee that nothing partial happened. A function that commits a database row, writes a file, and then throws has left two effects behind; a process killed between those steps leaves the same residue. What makes effects all-or-nothing is a transaction or an equivalent contract, never co-location in one process.

A synchronous function call normally reports completion or an exception through the same call stack. A process crash also stops the threads in that process, but it does not tell a restarted caller which external effects committed. Asynchronous work can separate the caller’s observation from execution even within one process: Python’s Future.result(timeout=...) can raise a timeout while the running task continues, and canceling a running future can fail. A network adds independently failing request and response paths, making it possible for the callee to complete while the caller learns nothing about the result.

The difficulty is not that components fail. It is that the caller frequently cannot tell which failure occurred. A request that produces no response within the timeout may have been lost on the way out, may have been processed successfully with the reply lost on the way back, or may still be executing. These require opposite responses — retry, do not retry, wait — and the information needed to choose is precisely what is missing.

A timeout does not resolve this. It converts an unbounded wait into a decision point, which is necessary, but the decision is still being made without knowing the outcome. Treating a timeout as evidence that nothing happened is the most common way duplicate charges and duplicate records get created.

Making an operation idempotent can make retries safe, but the contract must cover the same logical operation and a stable target. Setting a value to 100 twice is idempotent; replaying that older assignment after a newer assignment of 80 still loses the newer result. Request deduplication, version checks, or conditional updates may therefore be needed, with a stated retention window. Reconciliation can sometimes resolve an uncertain outcome; where it cannot, the system needs an explicit recovery policy.

The same reasoning applies to anything crossing a process boundary, which in a data platform includes every write to object storage, every call to a catalog, and every message published. A pipeline step that assumes its last write either fully succeeded or fully failed is assuming away the condition it operates under.

What to do about it — timeouts, retries, backoff, and the coordination that survives it — is worked through in Partial Failure, Timeouts, and Retries.

References

• Python: concurrent.futures — Future.result and Future.cancel