Stateful Stream Processing

Stateful stream processing is processing in which the result for one record depends on records seen earlier, so the job must keep something between records. A stateless transformation — parse this field, drop that one, route by type — remembers nothing and can be restarted anywhere. Almost everything else remembers.

The distinction matters because state is the part you operate. Descriptions of documented mechanics below follow Apache Flink’s documentation, checked in September 2026.

What a job remembers

• Window aggregates. A count or sum per key for every window that is still open — including windows held open by an allowed lateness horizon.
• Last-value state. The current status of each order, the latest reading per device — one entry per key, for as many keys as exist.
• Join buffers. One side of a stream-to-stream join held while waiting for the other, bounded by whatever time window the join allows.
• Deduplication sets. Identifiers already seen, retained for the replay horizon the pipeline supports.
• Source positions. Where each input had been read to, which must agree with the rest of the state for a restart to be correct.

Read that list and a pattern appears: every item has a retention rule, explicit or not. State that nobody decided to expire grows for as long as the job runs.

How it survives failure

State has to outlive the process holding it, which is what checkpoints provide. Flink’s description of the purpose is exact: checkpoints “make state in Flink fault tolerant by allowing state and the corresponding stream positions to be recovered, thereby giving the application the same semantics as a failure-free execution.” The two halves — state and the positions it corresponds to — are what make a restart consistent rather than merely possible.

Taking a consistent picture without stopping the stream uses barriers. When a checkpoint begins, the sources record their offsets and “insert numbered checkpoint barriers into their streams,” which then “flow through the job graph, indicating the part of the stream before and after each checkpoint.” An operator with two inputs performs barrier alignment so the snapshot reflects consumption “up to (but not past) both barriers.” Writing the state out need not block processing either: state backends “use a copy-on-write mechanism to allow stream processing to continue unimpeded while older versions of the state are being asynchronously snapshotted.”

Three operational consequences

• State size is recovery time. A restart restores state before processing resumes, so a job holding tens of gigabytes per instance does not come back in seconds. Convert the figure into a duration by timing an actual restore, and compare it with the recovery target the service has promised — the two are often set independently and discovered to disagree during an incident.
• Checkpoint duration is a symptom. Because barriers must align, one slow or skewed input holds up the whole checkpoint. Rising checkpoint times usually indicate a problem upstream of wherever the alert is configured, which makes the metric a useful early signal rather than a constant to ignore.
• State grows from decisions, not traffic. A longer lateness horizon, a wider join window, a deduplication set with no expiry, a key space that never retires — each is a choice someone made, and each adds state permanently. When memory grows without bound, the cause is usually a missing retention rule rather than unexpected volume.

What the guarantee covers

The strongest claim a stateful engine makes is narrower than it sounds. Flink’s phrasing for exactly-once is that “every event will affect the state being managed by Flink exactly once” — state, not the outside world. A record may be read twice after a restart; what the engine promises is that its own counters end up as if it had been read once. Anything written outward needs its own mechanism, either idempotent writes or a transaction committed with the checkpoint, as covered in two-phase commit sink.

How state fits with windows, watermarks, and end-to-end correctness is worked through in Stream Processing Semantics.

Reference: Apache Flink Documentation, Learn Flink: Fault Tolerance via State Snapshots; Apache Flink Documentation, Checkpoints.