Stream Processing

Stream processing handles data that keeps arriving, producing or updating results without waiting for the entire input to end. The logical input can be unbounded even when the engine processes records in small batches. Streaming does not mean zero delay.

Which five minutes does an event belong to?

Suppose a payment occurred at 10:04 but reached the processor at 10:07. Its event time is 10:04; its processing time is 10:07. A total for events in the interval from 10:00 inclusive to 10:05 exclusive should associate it with that interval if the calculation uses event time.

A window groups records into such finite intervals. A watermark estimates progress through event time, helping decide when to emit or finalize a result. It is not proof that no older record can arrive. These concepts are described in Apache Beam’s processing model.

What happens after an early result?

If a total was already shown before the delayed payment arrived, the system needs an explicit policy: revise the total, send a correction elsewhere, or exclude arrivals beyond a stated limit. Users need to know whether a displayed result is provisional.

The processor may retain state, such as a running total per store. Recovery must account for that state and for repeated records. If incoming work exceeds processing capacity, a backlog grows and results become older. Choosing streaming therefore does not remove the need to measure delay or define correctness.

Reference: Apache Beam: Basics of the model.