Incremental Extraction Watermark

An incremental extraction watermark is a stored position used to choose the next portion of source data. It may be a reliably maintained update timestamp or a sequence. After a successful load, the pipeline records the position it has covered. This is different from an event-time watermark used to decide progress in streaming windows.

Choose a fixed upper bound for a run, extract the agreed interval, and advance the checkpoint after destination commit. Advancing it first can lose data after a failed write. If the destination commits but checkpoint saving fails, the next attempt may repeat rows; use a retry-safe destination operation.

A timestamp can be shared by several rows. A timestamp-and-key cursor can order a stable extraction, but cannot discover a later commit carrying an older timestamp. Overlapping reads help only when paired with deduplication or version-aware upserts and a justified bound on lateness. Otherwise use a source mechanism with stronger change tracking or periodic reconciliation.

Reading current rows after a timestamp normally cannot see hard-deleted rows. Use deletion markers, a change feed, or comparison of complete snapshots when deletions matter. Record the initial-load boundary and recovery policy too; a watermark alone is not a history of every intermediate change.

Reference: Microsoft: incremental copy. See Designing End-to-End Data Architecture for the worked example.