Three Streaming Engines, Three Answers: Spark, Flink, and Kafka Streams

Apache Spark Structured Streaming, Apache Flink, and Kafka Streams all implement the same ideas: event time, windows, watermarks, state, and some claim about correctness. They differ in how those ideas are executed, where state lives, and — the part that usually decides the choice — what running the thing asks of the organization. Product behavior below follows Apache Spark 4.2.0, Apache Flink stable, and Apache Kafka 4.3.X documentation as checked in September 2026; execution modes and defaults change between versions, so verify against the version you will actually run.

The execution model sets the latency floor

Spark’s default engine treats a stream as a sequence of small batch jobs — micro-batch processing — and the documentation is unusually direct about the consequence. Continuous processing, its alternative mode, “enables low (~1 ms) end-to-end latency with at-least-once fault-tolerance guarantees,” and the guide asks you to “compare this with the default micro-batch processing engine which can achieve exactly-once guarantees but achieve latencies of ~100ms at best.” Continuous processing is also described as experimental, introduced in Spark 2.3.

Read that sentence twice, because it states a trade that marketing usually hides: within one product, the fast mode and the strong-guarantee mode are different modes. Flink and Kafka Streams take the other approach to execution — records flow through operators continuously, so their latency floor is not set by a batch interval.

They share that execution model and not the mechanism behind their guarantees, which is worth separating because the recovery sections below turn on it. Flink takes distributed snapshots of operator state, coordinated by barriers travelling with the records. Kafka Streams has no equivalent global snapshot: it leans on Kafka transactions to commit the input offsets, the changelog writes that back its state stores, and the output records as one unit, and it restores state by replaying the changelog. Same shape of execution, different answers to “what happens after a crash.”

The practical question is therefore not “which is fastest” but “what latency does the use case actually require.” A fraud check that must answer within 200 milliseconds and an hourly aggregate that must be ready by five past the hour are different problems, and only one of them is affected by any of this.

Where state lives, and what that costs

Every stateful job has to keep its aggregates somewhere, and the state backend is where the three products differ most concretely.

Engine	Where state is kept	Size limit
Flink, HashMapStateBackend	“Data internally as objects on the Java heap”	“Limited by available memory within the cluster”
Flink, EmbeddedRocksDBStateBackend	“In-flight data in a RocksDB database that is (per default) stored in the TaskManager local data directories”	“Only limited by the amount of disk space available”
Kafka Streams	Local state stores on the instance, each backed by “a replicated changelog Kafka topic in which it tracks any state updates”	Local disk, with the topic as the durable copy
Spark Structured Streaming	A state store maintained by the query, checkpointed to the configured location	Bounded by executor memory and the store implementation in use

Flink’s documentation states the trade in the RocksDB case plainly: it allows “keeping very large state,” but “each state access and update requires (de-)serialization and potentially reading from disk which leads to average performance that is an order of magnitude slower than the memory state backends.” An order of magnitude is not a tuning detail. It is the difference between a design that holds a hundred gigabytes of state per instance and one that must not.

Kafka Streams answers the durability question differently, and the answer is characteristic of it: state lives locally, and the log is the backup. Each store “maintains a replicated changelog Kafka topic,” partitioned so each local store has its own partition — a changelog topic. Recovery means replaying that topic into a fresh local store, which is simple to reason about and slow in proportion to how much state there is.

Recovery: the number nobody asks for until an incident

All three recover by restoring state and resuming from a recorded position; the mechanisms are variations of checkpoint and replay. What differs is how long that takes and what can shorten it.

• Flink restores from a checkpoint, which contains state and the corresponding source positions. Restore time scales with state size, and the state backend decides whether that state is a heap to rebuild or a local database to load.
• Kafka Streams replays the changelog topic — unless a standby replica exists, in which case the documentation says it “will assign a task to an application instance where such a standby replica already exists in order to minimize the task (re)initialization cost.” Standby replicas cost memory and disk on other instances and buy recovery time, which is a rare example of a dial whose two ends are both stated plainly.
• Spark resumes a query from its checkpoint location, re-processing the micro-batch that was in flight. Because progress is tracked through offset and commit logs, the checkpointing itself adds latency — which is why the engine offers asynchronous progress tracking, described as letting queries “checkpoint progress asynchronously and in parallel to the actual data processing within a micro-batch.” Read its limitations before reaching for it: the feature “is only supported in stateless queries using Kafka Sink,” and “exactly once end-to-end processing will not be supported with this asynchronous progress tracking because offset ranges for batch can be changed in case of failure.” It is a latency option for a narrow shape of query, not a general improvement to stateful recovery — and it is a different mechanism from the state store’s own checkpointing.

The question to take into a design review is the same in all three cases and is almost never answered in advance: with the state this job will hold in a year, how long does a restart take, and has anyone timed it?

Parallelism, rescaling, and upgrades

Kafka Streams ties parallelism directly to the input: “the maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by maximum number of partitions of the input topic(s).” That is a clean model with a hard ceiling — to scale beyond it you have to change the topic partition count, which has its own consequences for ordering. Flink and Spark decouple job parallelism from source partitions, at the cost of an exchange that redistributes records, with the data skew risks that come with any redistribution.

Changing the parallelism of a running stateful job is the harder problem, because state is partitioned by key and redistributing it means moving it. Flink’s answer is the savepoint: an operator-triggered snapshot you own, used to stop a job and start it again with different parallelism or upgraded code. Kafka Streams rebalances tasks across instances and rebuilds state from the changelog.

Spark needs the question split in two, because “restart from the checkpoint” covers only one half of it. Adding or removing executors changes how much compute is available and needs nothing special. Changing how the state itself is divided does not work that way: the number of state partitions is fixed by the shuffle partition setting at the time the query first ran, and the documentation lists it among the settings that cannot be changed for an existing checkpoint. Repartitioning state therefore means a new query with a new checkpoint, which brings its own plan — where to replay from, how to backfill, and how to hand consumers over from the old output without double-counting the overlap.

In every case the cost is proportional to state, which is the same sentence as the previous section for the same reason.

Upgrades deserve a mention because they are where teams discover that state has a schema. Changing the shape of what a job remembers — adding a field to an aggregate, changing a key — is a migration, not a deployment, whatever the engine. The engines differ in how much help they give and in nothing else.

Deployment shape is the organizational cost

Kafka Streams “is not a resource manager, but a library that ‘runs’ anywhere its stream processing application runs.” That single sentence is the most consequential difference in this comparison for most organizations. A Kafka Streams job is an application: it deploys like a service, scales like a service, and is operated by the team that owns the service. Flink is a cluster with a job manager and task managers to run, patch, and size — whether you run it yourself or buy it managed. Spark Structured Streaming runs on whatever Spark platform you already have, which is an advantage when one exists and a large commitment when it does not.

This is where a comparison based on features misleads most reliably. The engine that fits your problem best on paper can be the wrong choice if it adds a platform your team has no one to operate, and the total cost of ownership of a streaming platform is dominated by people rather than by licences.

Correctness claims, compared honestly

All three offer a strong guarantee with conditions attached, and the conditions are similar because the problem is.

• Spark: “Together, using replayable sources and idempotent sinks, Structured Streaming can ensure end-to-end exactly-once semantics under any failure” — with the caveat above that continuous mode provides at-least-once instead.
• Flink: exactly-once “means that every event will affect the state being managed by Flink exactly once,” and end to end requires that “your sources must be replayable, and your sinks must be transactional (or idempotent).”
• Kafka Streams: processing.guarantee accepts at_least_once, which is the default, and exactly_once_v2, which requires brokers at version 2.5 or higher and, by default, “a cluster of at least three brokers.”

Two things follow. The strong guarantee is usually not the default, so a platform running out of the box is running the weaker one — worth checking before assuming otherwise. And in every case the guarantee reaches only as far as systems that cooperate: a replayable source on one side, an idempotent or transactional sink on the other. No engine extends correctness into a destination that cannot participate, which is why a periodic reconciliation against the source belongs in the design regardless of which product is chosen.

Choosing

Condition	Points toward
Output is a table consumed by analytics, and the team already runs Spark	Structured Streaming — the marginal platform cost is near zero and the batch and streaming code share a model
Large keyed state, sub-second latency, complex event-time logic	Flink — the state backends and snapshot model are built for this, and the cluster is the price
The transformation belongs to one service, input and output are both Kafka topics	Kafka Streams — no platform to operate, and parallelism follows partitions
Latency requirement is minutes, not milliseconds	Whatever you already run, including a scheduled batch job — the streaming question may not arise at all

Three questions settle most of this before any product is evaluated. What latency does the decision downstream actually need, stated as a number with a consequence? How much state will this job hold at the volume you expect in a year, and how long does restoring it take? And who operates it at three in the morning — the team that owns the application, or a platform team that will need to exist first? The engines differ in interesting ways, but a choice made without those three answers is a preference dressed as an evaluation.

References: Apache Spark Documentation, Structured Streaming Programming Guide: Performance Tips and Getting Started; Apache Flink Documentation, State Backends and Learn Flink: Fault Tolerance via State Snapshots; Apache Kafka Documentation, Kafka Streams Architecture and Kafka Streams Configs. Checked September 2026; versions and defaults change.