Partial Failure

Partial failure is the condition in which some parts of a distributed system have failed while others continue working. It is the defining difference between a distributed system and a single program, where a crash stops everything at once and leaves no ambiguity about what happened.

The difficulty is not that components fail. It is that the caller frequently cannot tell which failure occurred. A request that produces no response within the timeout may have been lost on the way out, may have been processed successfully with the reply lost on the way back, or may still be executing. These require opposite responses — retry, do not retry, wait — and the information needed to choose is precisely what is missing.

A timeout does not resolve this. It converts an unbounded wait into a decision point, which is necessary, but the decision is still being made without knowing the outcome. Treating a timeout as evidence that nothing happened is the most common way duplicate charges and duplicate records get created.

Because the ambiguity cannot be removed, the designs that work are the ones that make it harmless. Making an operation idempotent — so that performing it twice has the same effect as once — means a retry is safe regardless of what actually happened, which sidesteps the question rather than answering it.

The same reasoning applies to anything crossing a process boundary, which in a data platform includes every write to object storage, every call to a catalog, and every message published. A pipeline step that assumes its last write either fully succeeded or fully failed is assuming away the condition it operates under.

What to do about it — timeouts, retries, backoff, and the coordination that survives it — is worked through in Partial Failure, Timeouts, and Retries.