Trigger and Accumulation Mode
In stream processing, a trigger decides when a window’s aggregate is emitted, and the accumulation mode decides what the next emission means relative to the previous one. Apache Beam states both precisely: triggers “determine when to emit the aggregated results of each window (referred to as a pane),” and “the accumulation mode determines whether the system accumulates the window panes as the trigger fires, or discards them.”
This is a different idea from an Airflow trigger rule, which decides whether a downstream task may run given its upstream task states. Here the question is not whether work starts but when a result is published.
Why a window emits more than once
A window over event time cannot be known to be complete; it can only be estimated as complete by a watermark. Waiting for that estimate gives one result per window and the longest delay. Emitting before it gives an earlier answer that may change. A trigger is how that choice is expressed, and Beam provides both event-time triggers, which fire relative to the watermark, and processing-time triggers, which fire on wall-clock progress regardless of what has arrived.
The default is the conservative one, and worth knowing before assuming a platform is unresponsive: with the default windowing configuration and default trigger, Beam “outputs the aggregated result when it estimates all data has arrived, and discards all subsequent data for that window.” One emission, nothing late. Anything more responsive is something you asked for.
The two modes, with the same firings
Beam’s worked example uses ten-minute windows and a trigger that fires every time three elements arrive. In accumulating mode, the successive firings for one key emit [5, 8, 3], then [5, 8, 3, 15, 19, 23], then [5, 8, 3, 15, 19, 23, 9, 13, 10] — each pane contains everything seen so far. In discarding mode, each firing emits only what arrived since the last one.
| Accumulating | Discarding | |
|---|---|---|
| Each emission contains | The window’s total so far | Only the delta since the previous emission |
| The consumer must | Replace the previous value for that window, having identified which emission is later | Combine the deltas — additive for sums and counts; for anything else, merge the partial state rather than adding the figures |
| A repeated emission | Harmless when the reader ends up reflecting the latest pane exactly once per key and window — a replace keyed by the aggregation key and the window where the later pane wins, or an append that carries key, window, and pane identifier and is read by selecting the latest within that scope. Not harmless when panes are indistinguishable, when an older pane can win, or when the key is the window alone and one group’s pane overwrites another’s | Double-counts unless the consumer deduplicates by pane |
| Reading one emission alone | Gives a valid, if incomplete, total | Gives a fragment that means nothing on its own |
Neither mode is more correct. What makes one wrong is a mismatch with what the destination does, and the table above states each pairing in its safe form rather than its convenient one. An upsert keyed by the aggregation key and the window together, or a dashboard reading the latest row for that pair, suits accumulating panes. The window alone is not a key — Beam defines an aggregation per key and window, and the 10:00 result for one store and the 10:00 result for another are different aggregates — and “latest” has to be decided by a pane version within that scope rather than by arrival, since a delayed earlier pane would otherwise overwrite a newer one. Session windows need one more thing: a late element can merge sessions already emitted, so the destination has to retract or replace the earlier rows, not only add the merged one. An append-only table that sums its rows suits discarding panes, provided the aggregate is one that can be added and each pane is counted once.
Two cases fall outside both defaults and are the ones that go wrong quietly. An aggregate that is not additive — an average, a distinct count, a ratio — cannot be assembled by adding discarding panes at all; what has to be combined is the partial state (the sum and the count, the set or sketch), not the emitted number. And an accumulating pane written by append leaves every partial figure in the table, so readers need the window and a pane identifier to pick the current one. The pairing is the design decision; the mode by itself is only half of it.
What consumers have to be told
Early emission means publishing numbers that will change, and that is a communication problem rather than a technical one. Two habits cover it. Mark a value as provisional where it is read, so nobody quotes a partial window as a final figure. And say when it stops changing — the point after which the number is fixed, which is a business decision rather than a property of the engine.
The two storage conditions above are the consumer’s side of the same contract: discarding output has to be combinable by whatever the consumer does with it, and accumulating output has to let the reader pick the latest emission for a given key and window. Beyond that, the consumer needs to know two things the storage layout does not say. “Latest known” is not “final” while lateness is still allowed. And with session windows, a late record can merge two sessions that were already published, so a session figure can disappear as well as change.
Where a figure changes after consumers have already acted on it, the change gets the treatment the agreed policy gives any corrected period: a provisional pane refreshing within its stated contract is expected and needs no notice, while a change to a figure presented as final is a restatement, announced or logged according to its size and kind. How triggers, accumulation, windows, and lateness settings combine into a result someone can rely on is worked through in Stream Processing Semantics; the grouping they operate on is covered in stream windowing.
Further reading
- Apache Beam Programming Guide, Triggers and window accumulation modes
- Basics of the Beam model, Aggregation
- Apache Flink Documentation, Windows: Late elements considerations
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.