State Backend

A state backend is the component that decides where and how a stream processing job stores the working state it keeps between records — window aggregates, last values per key, join buffers — and how that state is snapshotted for recovery. Apache Flink names the choice explicitly, and its two long-standing bundled options — heap and embedded RocksDB — make the trade easy to see. A third, ForStStateBackend, keeps state in remote storage with a local cache and is documented as experimental with unsupported features; it is left out of the comparison below because its costs fall differently, not because it does not exist.

Two places to keep it

The documentation summarizes the reason anyone accepts the slower option: it allows “keeping very large state, compared to the HashMapStateBackend that keeps state in memory.” That is the core of the decision between the two — how much state, against how fast each access has to be — with recovery behavior as the third term, below.

How each choice shows up in operation

• Heap-based state fails as memory pressure. Because the state is live objects, growth is paid in garbage collection: pauses lengthen, then latency becomes erratic, and eventually the job fails on memory rather than on state size. The symptom arrives before the cause is obvious, which is why the working-set estimate matters more than the failure threshold.
• Disk-based state makes local storage a requirement. State sits in the task manager’s local directories, so instance sizing now includes disk capacity and speed, and a node with slow local storage becomes a slow worker rather than an equal one.
• Both make restart cost depend on state, by different routes. A restarted job restores state before processing resumes — see checkpoint and replay — and how long that takes depends on what has to be transferred, whether valid local state already exists, whether the checkpoint is full or incremental (which the documentation notes can make a restore faster or slower depending on the bottleneck), and how quickly throughput recovers afterward. Time an actual restore for the backend you use rather than estimating it from size.

Deciding in the right order

The backend is usually chosen first and questioned later, which is backwards. Estimate the state first: how many keys are live at once, how much is kept per key, how long each entry stays — including entries held open by a lateness horizon or a join window. That number, not a product preference, is what selects the backend, and it also tells you whether the design needs a retention rule more than it needs a bigger machine.

What a job keeps state for, and why it grows from decisions rather than from traffic, is covered in stateful stream processing. How the three common engines answer the same question differently — including engines that keep state locally and back it with a log rather than a snapshot — is worked through in Three Streaming Engines, Three Answers.

Further reading

• Apache Flink Documentation, State Backends
• Apache Flink Documentation, Disaggregated State Management