Late Data and Lookback

Late data arrives after a relevant processing cutoff or progress boundary. Always state which boundary: ten minutes of delivery delay does not by itself say whether a record is too late for a window. A lookback is a policy of revisiting an earlier range so newly available records can revise its output.

Suppose a run at 10:10 recomputes the latest closed five-minute window [10:05, 10:10) and one preceding window [10:00, 10:05). A 10:02 sale arriving at 10:09 can now be included. A 09:58 sale cannot, because its window is outside this run’s horizon. Specify inclusion at both the arrival cutoff and event-window boundaries; a record arriving exactly at a cutoff may wait until the next run.

Replace the affected aggregate or apply a verified correction, rather than appending a full recalculated total. Retain enough source data to rebuild it. A changed value under an existing ID needs a version or correction rule, not just deduplication. Choose the horizon from observed delays, corrections and rebuild cost, and keep a targeted backfill route for exceptions. Matching a finite sample does not prove that future delays fit the horizon.

Reference: Apache Beam late data and triggers.