Incremental Processing

Incremental processing updates a result using a bounded change set and retained state rather than rebuilding the entire result. It may reread an overlap or recompute affected partitions.

Define what counts as a change, which clock or position selects it, and which outputs depend on it. A changed dimension can affect many facts. An update that changes a partition key may require repairs in both the old and new partitions.

A timestamp cursor does not automatically capture late commits or hard deletes. Use a supported source protocol and correction or reconciliation paths. A valid empty change interval differs from missing input.

Incremental work is not automatically idempotent or correct. Specify stable operation identity, retry-safe publication, and progress advancement. A full rebuild can be idempotent too, and may be needed when retained state or transformation logic changes.

See Building Incremental and Idempotent Pipelines for a transactional example.

Reference: Official documentation.