Savepoint
A savepoint is an operator-owned snapshot for controlled restarts and planned changes. Retained checkpoints can also support some changes, subject to compatibility limits.
-
-
State Backend
Heap state is limited by available memory. Embedded RocksDB supports larger state within available disk capacity, with serialization and storage costs on access.
-
Spark, Flink, and Kafka Streams Compared
The choice is rarely about features. It is about the latency you actually need, how much state you will carry, and whether you want a cluster to operate.
-
Two-Phase Commit Sink
A completed checkpoint records progress; a destination commit exposes output. The sink must recover safely when failure separates those two actions.
-
Stateful Stream Processing
A stateful job retains memory across events and failures. Recovery time depends on state size, the recovery path, and available resources.
-
Allowed Lateness
An engine that silently drops late records is not misbehaving. Keeping a window open to accept them costs state, which is why the default is to keep nothing.
-
Trigger and Accumulation Mode
Emitting early is easy. Deciding whether the second number replaces the first or is added to it is the part that determines whether consumers can use either.
-
Stream Windowing
An event-time window groups records by when their events occurred, rather than by when a job runs. Window shape and measure additivity determine how results can be combined.
-
Stream Processing Semantics and Correctness
A streaming result describes data received by an agreed boundary. See how windows, lateness, state, and publication rules make revisions understandable.
-
Lambda Architecture
The Lambda architecture pairs a batch path with a fast streaming path — and then has to keep two implementations of the same logic in agreement.
-
Kappa Architecture
Kappa keeps a single streaming path. To fix logic or recompute history, it replays a retained log into a new output and switches consumers when it catches up.
-
Operating Streams: Backpressure and Replay
Lag, replay, and upgrades are where stateful streaming jobs usually fail. Work through the arithmetic and the procedures that keep restarts and rollouts safe.
-
Event Time, Watermarks, and Late Data
A watermark is an estimate, not a fact. See how the delay you choose trades result latency against dropped or revised records, and what it costs in state.
-
Backpressure
Understand how slow downstream work limits upstream progress, calculate backlog growth, and distinguish buffering from extra processing capacity.
-
Checkpoint and Replay
Restore consistent operator state and input position, understand replay, and separate checkpoint recovery from exactly-once external effects.
-
Late Data and Lookback
Use a lookback to revise recent results, distinguish late arrivals from corrections, and plan a route for data beyond the automatic horizon.
-
Event-Time Watermark
Understand an event-time watermark, how it differs from an extraction cursor, and why delay, idle inputs, and bad clocks change window results.
-
Event Time and Processing Time
Distinguish when an event happened from when an operator handled it, choose the clock for a metric, and check timestamp quality before grouping.
-
Micro-Batch Processing
See how repeated bounded increments can power streaming, why trigger frequency is not freshness, and what safe retries must preserve.
-
Stream Processing
Learn how streaming handles ongoing input, event time, windows, and late arrivals without assuming zero delay.