Allowed Lateness

Allowed lateness is the period after a window’s watermark has passed during which the engine still keeps that window’s state, so a record arriving late can be included and the result revised. In Apache Beam it is set with withAllowedLateness — the guide’s example allows late data up to two days after the end of a window. Descriptions follow that guide, checked in September 2026.

The setting exists because the watermark is an estimate. Once it moves past the end of a window, any further element with a timestamp in that window is late data, and something has to decide what happens to it.

Three possible answers

• Drop it. Cheapest, and the default. Beam’s default windowing configuration “has an allowed lateness value of 0,” and its default behavior is to “discard late data.” A pipeline nobody configured therefore loses late records silently — not a malfunction, but the consequence of a setting left alone.
• Revise the window. Allowed lateness keeps the state so the aggregate can be recomputed. Beam notes that when lateness is allowed, “the default trigger will emit new results immediately whenever late data arrives” — so the revision is published, and consumers need to expect changing values.
• Route it elsewhere. Send late records to a separate output for inspection or batch correction. This keeps the streaming path’s state bounded while preserving the records, which matters where dropping them is not acceptable.

The cost is state, not compute

Keeping a window open means keeping its accumulated state, per key, for the whole lateness period. A two-day horizon on hourly windows keeps roughly forty-eight windows’ worth of state alive for every key that is still in play, and that memory has to be checkpointed and restored like any other state — which is why a generous lateness setting shows up later as slow recovery rather than as a memory error. The relationship is worked through in stateful stream processing.

So the horizon is a trade rather than a preference: longer means more late records included and more state carried; shorter means a leaner job that forgets sooner. Derive it from observed delays rather than from instinct — how late records actually arrive, over a period long enough to include a bad day — and treat matching a sample as evidence about that sample, not a guarantee about future delays.

One propagation rule catches teams out. Beam states that allowed lateness “propagates forward to any subsequent PCollection derived from the first PCollection you applied allowed lateness to,” and changing it later requires an explicit Window.configure().withAllowedLateness(). A value chosen for one stage therefore becomes the default for everything downstream of it, including stages where a different horizon was intended.

Beyond the horizon

Allowed lateness does not remove the need for a correction path; it decides where the boundary between the two lies. Inside the horizon, the streaming job revises its own output. Outside it, a record has to be handled by recomputing the affected period in batch and republishing it — which, where consumers have already seen the old numbers, is a restatement to announce rather than a quiet fix. Stating both the horizon and the correction route is what makes the pipeline’s behavior describable: records later than this are not in the streaming result, and here is what happens to them instead.

How lateness interacts with windows, triggers, and end-to-end correctness is worked through in Stream Processing Semantics.

Reference: Apache Beam Programming Guide, Managing late data.