Spark, Flink, and Kafka Streams Compared

Apache Spark Structured Streaming, Apache Flink, and Kafka Streams all implement the same ideas: event time, windows, a way of advancing time and handling late data — watermarks in Spark and Flink, stream time and grace periods in Kafka Streams — state, and some claim about correctness. They differ in how those ideas are executed, where state lives, and — the part that usually decides the choice — what running the thing asks of the organization. Product behavior below follows Apache Spark 4.2.0, Apache Flink stable, and Apache Kafka 4.3.X documentation; execution modes and defaults are version-dependent.

The execution model sets the latency floor

Spark’s default engine treats a stream as a sequence of small batch jobs — micro-batch processing — and the documentation is unusually direct about the consequence. The guide describes continuous processing as capable of roughly 1 ms end-to-end latency with at-least-once recovery, compared with roughly 100 ms at best for the default micro-batch engine with exactly-once semantics. Continuous processing is also described as experimental, introduced in Spark 2.3 — and the same guide limits what it can run: map-like operations such as projections and filters, over a restricted set of sources and sinks, with no aggregations. For a stateful aggregate it is therefore not an alternative at all; the choice between latency and guarantee arises only once the supported operations cover the job.

The sentence is worth dwelling on, because it states a trade that comparison charts rarely make explicit: within one product, the fast mode and the strong-guarantee mode are different modes, and the guide’s figures describe those modes rather than a benchmark of any workload or a floor every job will reach. Flink and Kafka Streams take the other approach to execution — records flow through operators continuously, so their latency floor is not set by a batch interval. That is not the same as a consumer seeing each result at once: Kafka Streams, for one, holds store updates in a record cache and forwards them on commit or when the cache fills, so output can leave in bursts.

They share that execution model and not the mechanism behind their guarantees, which is worth separating because the recovery sections below turn on it. Flink takes distributed snapshots of operator state, coordinated by barriers traveling with the records. Kafka Streams has no equivalent global snapshot: with processing.guarantee set to exactly_once_v2 it leans on Kafka transactions to commit the input offsets, the changelog writes that back its state stores, and the output records as one unit — under the default at_least_once it commits offsets after the writes and accepts reprocessing — and either way it restores state by replaying the changelog. The boundary of that guarantee is Kafka topics and Kafka-managed state. A write to an external database does not join the transaction however transactional the database is, so a failure between the database commit and the Kafka commit re-executes the external work on restart; such effects need their own idempotent or atomic protocol. Same shape of execution, different answers to “what happens after a crash.”

The practical question is therefore not “which is fastest” but “what latency does the use case actually require.” A fraud check that must answer within 200 milliseconds and an hourly aggregate that must be ready by five past the hour are different problems, and the execution model touches each differently: the fraud check has to be measured to the moment its answer is visible downstream, buffering and commits included, while the hourly aggregate is more exposed to watermark progress and recovery delay than to the per-record latency floor.

Where state lives, and what that costs

Every stateful job has to keep its aggregates somewhere, and the state backend is where the three products differ most concretely.

Engine	Where state is kept	Size limit
Flink, HashMapStateBackend	“Data internally as objects on the Java heap”	“Limited by available memory within the cluster”
Flink, EmbeddedRocksDBStateBackend	“In-flight data in a RocksDB database that is (per default) stored in the TaskManager local data directories”	“Only limited by the amount of disk space available”
Kafka Streams	Local state stores on the instance, each backed by “a replicated changelog Kafka topic in which it tracks any state updates”	Local disk, with the topic as the durable copy
Spark Structured Streaming	A state store per partition, kept by the default HDFS-backed provider as in-memory maps on the executors or by the RocksDB provider in native memory and local disk, and checkpointed to the configured location either way	Executor memory with the default provider; native memory and local disk with RocksDB, which is the provider a large-state query is expected to use

Flink’s documentation states the trade in the RocksDB case plainly: it allows “keeping very large state,” but “each state access and update requires (de-)serialization and potentially reading from disk which leads to average performance that is an order of magnitude slower than the memory state backends.” The extra state capacity trades against per-access cost: a job holding a hundred gigabytes per instance needs enough disk and throughput for its workload.

Kafka Streams answers the durability question differently, and the answer is characteristic of it: state lives locally, and the log is the backup. Each store “maintains a replicated changelog Kafka topic,” partitioned so each local store has its own partition — a changelog topic. Recovery means replaying that topic into a local store — from the beginning when no usable local copy exists, from where the local copy left off when one does — which is simple to reason about and takes time in proportion to how much changelog has to be replayed; compaction keeps that close to the size of the state only once the cleaner has caught up.

Recovery: the number nobody asks for until an incident

All three recover by restoring state and resuming from a recorded position; the mechanisms are variations of checkpoint and replay. What differs is how long that takes and what can shorten it.

• Flink restores from a checkpoint, which contains state and the corresponding source positions. Restore cost depends on how much state has to be moved and on the backend: a heap backend rebuilds objects from the snapshot; RocksDB loads a local database, faster or slower with incremental checkpoints depending on where the bottleneck is; and the newer ForSt backend, which keeps state in remote storage and is still marked experimental, can resume without downloading everything, paying in remote reads afterwards. Time a restore for the backend you use rather than deriving it from size.
• Kafka Streams replays the changelog topic — unless a standby replica exists, in which case the documentation says it “will assign a task to an application instance where such a standby replica already exists in order to minimize the task (re)initialization cost.” A standby still has to apply whatever it had not yet caught up on, so it shortens the replay rather than removing it. Standby replicas cost memory and disk on other instances and buy recovery time, which is a rare example of a dial whose two ends are both stated plainly.
• Spark resumes a query from its checkpoint location, re-processing the micro-batch that was in flight. Because progress is tracked through offset and commit logs, the checkpointing itself adds latency — which is why the engine offers asynchronous progress tracking, described as letting queries “checkpoint progress asynchronously and in parallel to the actual data processing within a micro-batch.” Read its limitations before reaching for it: the feature supports only stateless queries using the Kafka sink and does not provide end-to-end exactly-once semantics, because failure can change the batch’s offset ranges. It is a latency option for a narrow shape of query, not a general improvement to stateful recovery — and it is a different mechanism from the state store’s own checkpointing.

The question to take into a design review is the same in all three cases and is best answered before an incident asks it: with the state this job will hold in a year, how long does a restart take, and has anyone timed it?

Parallelism, rescaling, and upgrades

Kafka Streams ties parallelism directly to the input: “the maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by maximum number of partitions of the input topic(s).” That ceiling applies per sub-topology, and there are two ways past it: add partitions to the source topic, or repartition inside the application through an internal topic whose partition count you choose, so that everything downstream of it runs at the higher parallelism while the source side keeps reading at its own. Both change which keys land where, with consequences for ordering and for existing state, so neither is a resize. Flink and Spark decouple job parallelism from source partitions, at the cost of an exchange that redistributes records, with the data skew risks that come with any redistribution.

Changing the parallelism of a running stateful job is the harder problem, because state is partitioned by key and redistributing it means moving it. Flink’s answer is a snapshot to restart from — usually the savepoint, an operator-triggered snapshot you own, taken to stop a job and start it again with different parallelism or upgraded code, though a retained checkpoint can serve for rescaling and for upgrades that keep the state readable. Either way the snapshot holds only the state: whether the new code can read it, with stable operator identifiers and compatible serializers, is a separate check, and restarting from an older snapshot does not undo output already committed, so a fork or a rollback needs its own sinks or an agreed policy for writes applied again. Kafka Streams rebalances tasks across instances and rebuilds state from the changelog.

Spark needs the question split in two, because “restart from the checkpoint” covers only one half of it. Adding or removing executors changes how much compute is available and is not an operation on the checkpoint — though it is not free: a task that lands on an executor without the state it needs loads that state from the checkpoint first, so moving executors around costs state locality. Changing how the state itself is divided does not work that way: the number of state partitions is fixed by the shuffle partition setting at the time the query first ran, and the documentation lists it among the settings that cannot be changed for an existing checkpoint. Repartitioning state therefore means a new query with a new checkpoint, which brings its own plan — where to replay from, how to backfill, and how to hand consumers over from the old output without double-counting the overlap.

In every case the cost rises with state, but not by one formula: what has to be transferred, whether a valid local copy already exists, the format being restored, and how long it takes to recover full throughput all enter — the same lesson as the previous section, for the same reason.

Upgrades deserve a mention because they are where teams discover that state has a schema. Changing the shape of what a job remembers — adding a field to an aggregate, changing a key — is a migration, not a deployment, whatever the engine. The engines differ in the mechanics — a snapshot, a changelog, a checkpoint location — and share the requirement that state compatibility is verified before the new version meets real state.

Deployment shape is the organizational cost

Kafka Streams is a library inside the streaming application rather than a separate resource manager. That single sentence is the most consequential difference in this comparison for most organizations. A Kafka Streams job is an application: it deploys like a service, scales like a service, and is operated by the team that owns the service. What it removes is a separate processing cluster, not the platform: the brokers, the application fleet, the local stores, and the changelog topics are still someone’s to run. Flink is a cluster with a job manager and task managers to run, patch, and size — whether you run it yourself or buy it managed, which moves much of that work to the vendor but leaves the state, upgrade, and cost questions with you. Spark Structured Streaming runs on whatever Spark platform you already have, which is an advantage when one exists — though a batch-only platform still has to add always-on capacity, checkpoint storage, and someone on call before it is a streaming platform — and a large commitment when it does not.

This is where a comparison based on features misleads most reliably. The engine that fits your problem best on paper can be the wrong choice if it adds a platform your team has no one to operate, and the total cost of ownership of a streaming platform includes the people to run it as well as the compute, storage, and network it consumes. Compare all of them for the same workload, period, and scope rather than assuming in advance which one dominates.

Correctness claims, compared honestly

All three offer a strong guarantee with conditions attached, and the conditions are similar because the problem is.

• Spark: “Together, using replayable sources and idempotent sinks, Structured Streaming can ensure end-to-end exactly-once semantics under any failure” — with the caveat above that continuous mode provides at-least-once instead.
• Flink: exactly-once “means that every event will affect the state being managed by Flink exactly once,” and end to end requires that “your sources must be replayable, and your sinks must be transactional (or idempotent).”
• Kafka Streams: processing.guarantee accepts at_least_once, which is the default, and exactly_once_v2, which requires brokers at version 2.5 or higher and, by default, “a cluster of at least three brokers.”

Two things follow. What is on by default differs by product and is not one switch: Spark’s micro-batch engine gives exactly-once only with a replayable source and an idempotent or transactional sink, and continuous mode gives at-least-once; Flink needs checkpointing enabled, in the right mode, with connectors that support it; Kafka Streams runs at-least-once until processing.guarantee is changed. Check each condition for the version you run rather than assuming the strong guarantee is on — or off. And in every case the guarantee reaches only as far as systems that cooperate: a replayable source on one side, an idempotent or transactional sink on the other. No engine extends correctness into a destination that cannot participate, which is why a periodic reconciliation against the source belongs in the design regardless of which product is chosen.

Choosing

Condition	Points toward
Output is a table consumed by analytics, and the team already runs Spark	Structured Streaming — the platform, tooling, and skills are already paid for, so the added cost is the always-on capacity, checkpoint storage, and on-call that streaming brings, and the batch and streaming code share a model
Large keyed state, sub-second latency, complex event-time logic	Flink — the state backends and snapshot model are built for this, and the cluster is the price
The transformation belongs to one service, input and output are both Kafka topics	Kafka Streams — no separate processing cluster to operate, since the brokers and the service platform are already there, and parallelism follows partitions
Latency requirement is minutes, not milliseconds	Whatever you already run, including a scheduled batch job — the streaming question may not arise at all

Three questions settle most of this before any product is evaluated. What latency does the decision downstream actually need, stated as a number with a consequence? How much state will this job hold at the volume you expect in a year, and how long does restoring it take? And who operates it at three in the morning — the team that owns the application, or a platform team that will need to exist first? The engines differ in interesting ways, but a choice made without those three answers is a preference dressed as an evaluation.

References

• Apache Spark Documentation, Structured Streaming Programming Guide: Performance Tips (continuous processing, supported queries, asynchronous progress tracking)
• Apache Spark Documentation, Structured Streaming Programming Guide: Getting Started (fault-tolerance semantics)
• Apache Spark Documentation, Structured Streaming Programming Guide: State Store
• Apache Spark Documentation, Structured Streaming Programming Guide: Additional Information (settings that cannot change across restarts)
• Apache Flink Documentation, State Backends
• Apache Flink Documentation, Disaggregated State Management
• Apache Flink Documentation, Learn Flink: Fault Tolerance via State Snapshots
• Apache Flink Documentation, Savepoints
• Apache Flink Documentation, Checkpoints vs. Savepoints
• Apache Kafka Documentation, Kafka Streams Architecture
• Apache Kafka Documentation, Kafka Streams Core Concepts (time, processing guarantees)
• Apache Kafka Documentation, Kafka Streams DSL (repartition)
• Apache Kafka Documentation, Kafka Streams Memory Management
• Apache Kafka Documentation, Kafka Streams Configs