State Backend
A state backend is the component that decides where and how a stream processing job stores the working state it keeps between records — window aggregates, last values per key, join buffers — and how that state is snapshotted for recovery. Apache Flink names the choice explicitly, and its two bundled options make the trade easy to see. Descriptions follow Flink’s documentation, checked in September 2026.
Two places to keep it
HashMapStateBackend | EmbeddedRocksDBStateBackend | |
|---|---|---|
| Where state lives | “Data internally as objects on the Java heap” | “In-flight data in a RocksDB database that is (per default) stored in the TaskManager local data directories” |
| Size limit | “Limited by available memory within the cluster” | “Only limited by the amount of disk space available” |
| Cost per access | An object lookup | “(De-)serialization and potentially reading from disk,” which the documentation describes as “an order of magnitude slower than the memory state backends” |
The documentation summarizes the reason anyone accepts the slower option: it allows “keeping very large state, compared to the HashMapStateBackend that keeps state in memory.” That is the whole decision — how much state, against how fast each access has to be.
How each choice shows up in operation
- Heap-based state fails as memory pressure. Because the state is live objects, growth is paid in garbage collection: pauses lengthen, then latency becomes erratic, and eventually the job fails on memory rather than on state size. The symptom arrives before the cause is obvious, which is why the working-set estimate matters more than the failure threshold.
- Disk-based state makes local storage a requirement. State sits in the task manager’s local directories, so instance sizing now includes disk capacity and speed, and a node with slow local storage becomes a slow worker rather than an equal one.
- Both make restart time a function of state size. Whatever the backend, a restarted job restores state before processing resumes — see checkpoint and replay — and the honest way to know that duration is to time an actual restore rather than to estimate it.
Deciding in the right order
The backend is usually chosen first and questioned later, which is backwards. Estimate the state first: how many keys are live at once, how much is kept per key, how long each entry stays — including entries held open by a lateness horizon or a join window. That number, not a product preference, is what selects the backend, and it also tells you whether the design needs a retention rule more than it needs a bigger machine.
What a job keeps state for, and why it grows from decisions rather than from traffic, is covered in stateful stream processing. How the three common engines answer the same question differently — including engines that keep state locally and back it with a log rather than a snapshot — is worked through in Three Streaming Engines, Three Answers.
Reference: Apache Flink Documentation, State Backends.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.