Event Deduplication

Event deduplication recognizes repeated deliveries of the same logical event so that downstream processing can avoid counting it again. Generate a stable event ID when the event is created and preserve it on retry — the producing side of the same identity rule that makes a write idempotent. Where events are published from a database change, a transactional outbox is a common place to assign that ID. Deduplicating equal-looking rows without an identity rule can erase legitimate repeated actions.

Two deliveries with one ID may differ in transport timestamps yet carry the same business payload. Compare the logical fields under a documented policy. If the same ID has two different amounts, quarantine or reject the conflict instead of keeping whichever arrived first — see data quarantine for holding such records where they can be examined rather than dropped.

The state itself is a set of identifiers already seen, keyed by event ID and expired after a chosen horizon — which is why the horizon, not the algorithm, is the design decision. Segment’s documentation is a useful concrete case: it deduplicates “on the event’s messageId, not on the contents of the event payload,” stores “at least 24 hours’ worth of event messageIds,” and states the guarantee as probabilistic — 99 percent of data free of duplicates “within an approximately 24 hour look-back window” — with longer-range duplicates handled later in the warehouse. Two things follow for anyone building this: a duplicate that arrives after the horizon passes through, and the horizon is a storage cost, so it is chosen rather than maximized. Retain deduplication state for the supported retry and replay horizon — which includes deliberate reprocessing, not only transport retries, so the horizon should be compared with how far back a replay can start.

Coordinate its update with the intended effect. A check followed by an uncoordinated write can race. Two different event IDs for one order require a separate business-key rule; one order can also legitimately have several event types. Removing duplicates from a table does not by itself prevent repeated external emails.

Further reading

• Segment duplicate handling