Checkpoint and Replay

A checkpoint preserves a recoverable point in a computation. For a stateful stream this includes coordinated operator state and source progress; the exact contents depend on the engine and checkpoint mode. Replay reads retained input again from that point to reconstruct later state, which means the input must still be there: a retention window shorter than the outage turns recovery into re-seeding from a snapshot rather than a replay. A position without the corresponding aggregate is not enough.

The part that sounds impossible is taking a consistent picture of a running computation without stopping it, and the standard answer is worth knowing because it explains the failure modes. Flink’s documentation describes it as barriers: when a checkpoint begins, the sources “record their offsets and insert numbered checkpoint barriers into their streams,” and “these barriers flow through the job graph, indicating the part of the stream before and after each checkpoint.” Each operator records its state as the barrier reaches it, and an operator with two inputs performs barrier alignment “so that the snapshot will reflect the state resulting from consuming events from both input streams up to (but not past) both barriers.” Writing the state out does not have to block the stream either — Flink’s state backends “use a copy-on-write mechanism to allow stream processing to continue unimpeded while older versions of the state are being asynchronously snapshotted.” That is the aligned form of the mechanism; Flink also offers unaligned checkpoints, in which a barrier can overtake buffered records and the checkpoint stores those in-flight records instead of waiting for them to be processed — quicker to complete under backpressure, larger to store and restore.

Two things follow from that design. A slow or skewed input holds up alignment, so checkpoint duration is a symptom worth watching rather than a fixed cost. And the guarantee is narrower than it sounds: Flink’s own phrasing is that exactly once means “every event will affect the state being managed by Flink exactly once” — state, not the outside world, which is why the paragraph below about the already-sent email is not a caveat but the main limitation.

The same problem without a streaming engine

Outside a stateful engine the mechanism is cruder and the choice is starker: a job writes output and records how far it got, and those are two separate writes. Which one goes first decides which failure you get. Take a batch that reads input records 100–199.

Writing output first avoids advancing normal recovery past output that was never committed, but it creates a replay window. A position-first gap is not inherently undetectable: retained input, a manifest, or key-level reconciliation can identify and recover missing records such as 100–199. Recovery then needs an explicit rewind or repair rather than blindly resuming at the stored position. Without retained input or evidence of what is missing, repair may be impossible. Common ways to coordinate progress and output include:

• Commit them atomically. A transaction containing both output and progress leaves both committed or neither. Storing progress in the destination database is one implementation. Other arrangements need an equivalent atomic commit or publication protocol covering both; independently writing a file beside a database does not provide it.
• Make replay safe. Write output first to a destination that recognizes repeated logical operations or applies state under an appropriate version guard, then record progress. A key alone does not prevent replay of old state from overwriting newer state. Input must remain available, and the replay contract must cover the effects being recovered.

Note what the second option does not cover: a repeat is harmless in the destination, not in anything the job did on the way — a file emitted to a partner, a message published, a notification sent. Those need their own de-duplication, which is the same boundary the engine-level guarantee has.

If a checkpoint records state after records 0 through 999 and the next position is 1000, a crash before record 1234 means replay starts at 1000. The 234 already processed records are recomputed from the restored state, not added to the lost state a second time. This requires the input still to exist and compatible code and reference data. External lookups or clocks can make replay nondeterministic — the same hazard that makes a rerun of a batch job produce different output than the original run.

An email sent before failure is not undone by restoring a checkpoint. End-to-end exactly-once effects require a compatible sink protocol or replay-safe operations within a stated boundary — in practice, idempotent writes that recognize the same operation and preserve newer state during replay. Nor does recovery repair data that was wrong rather than missing; correcting a range is a backfill. Nor does recovery recover events intentionally dropped by the lateness policy: it reproduces that policy.

One operational detail catches teams out: checkpoints belong to the engine, not to you. Flink deletes them when a job ends unless retention is configured, and the retention setting distinguishes cancellation from failure — so a job that was canceled rather than crashed can leave nothing to restore from. A point you intend to restart from deliberately is a savepoint, which is triggered and owned by the operator instead. Test restoration, input retention, sink behavior and state compatibility, rather than treating a successful checkpoint write as proof of recovery.

Further reading

• Flink checkpoints
• Apache Flink Documentation, Fault Tolerance via State Snapshots
• Apache Flink Documentation, Checkpointing under backpressure