Incremental Snapshot
An incremental snapshot copies the existing contents of a table in chunks while change data capture continues streaming, instead of doing one uninterrupted scan before streaming starts. It exists because a transaction log does not hold a database’s full history — Netflix’s DBLog paper states the problem as needing to replicate the full state alongside the log — and because the obvious way to get that state, one long blocking scan, is costly on a busy source.
The watermark mechanism
The difficulty is that rows selected from the table and events arriving from the log describe overlapping time. The DBLog approach resolves the overlap explicitly, per chunk:
- Pause log processing briefly.
- Write a low watermark by updating a dedicated watermark table, so it appears in the log itself.
- Select the chunk into memory.
- Write a high watermark the same way.
- Resume log processing, sending log events onward as usual.
- When the low watermark event arrives in the stream, start removing from the in-memory chunk any rows whose primary keys change before the high watermark arrives.
- When the high watermark event arrives, append whatever remains of the chunk to the output.
The rule this enforces is that a log event always beats a selected row for the same key, which satisfies the paper’s stated requirement that an older version of a row is not delivered after a later version. Debezium documents an equivalent facility, comparing primary keys of buffered snapshot events against streamed events within a snapshot window and discarding the superseded snapshot records.
Compared with a blocking snapshot
| Blocking snapshot | Incremental snapshot | |
|---|---|---|
| Streaming during the copy | Starts after the snapshot completes | Continues throughout |
| Interruption | Per Debezium’s documentation, the default PostgreSQL snapshot restarts from the beginning | Resumes from the last completed chunk |
| Source impact | One long-running read, with a transaction held open for its duration | Many bounded reads; the paper notes the watermark approach uses no locks and has minimum impact on the source |
| Requirements | A consistent read and the matching log position | A key to chunk on, and write access for the watermark table where the method needs one |
What it changes operationally
Because chunked snapshots can be triggered at any time — the paper describes running them for all tables, one table, or specific primary keys — re-seeding stops being an outage. A table corrupted downstream, a column added to the destination, or a consumer that fell behind log retention can all be repaired by snapshotting that table again while the stream keeps flowing. In effect, seeding and targeted backfill become the same operation.
What it does not change is the requirement that the snapshot and the stream meet at one well-defined boundary; it only moves that boundary from a single instant to a per-chunk rule. See snapshot and stream handoff for the requirement itself, and Incremental Loading and CDC: How a Pipeline Knows What Changed for how the first load fits into a change-capture pipeline.
References: Andreas Andreakis and Ioannis Papapanagiotou, DBLog: A Watermark Based Change-Data-Capture Framework, Netflix, 2020; Debezium documentation, Debezium connector for PostgreSQL.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.