Replica Identity
Whether a delete event tells you only the key or the whole row is a source-side setting. It decides what history and audits downstream can reconstruct.
-
-
Log Sequence Number (LSN)
A log sequence number points at a place in the transaction log. It is how a pipeline knows the order of changes, where to resume, and what to ignore.
-
Incremental Snapshot
Seeding a table no longer has to mean one long locked scan. Chunks and watermarks let a snapshot run beside the change stream, and pause and resume.
-
Incremental Loading and CDC
A nightly job filtering on updated_at looks right until rows quietly go missing. Here is how each change-capture method works, and what none of them can see.
-
ETL vs ELT, Batch vs Streaming
“We need a pipeline” hides several decisions. See when to copy data or call its owner, where to transform, how fresh to be, and who pays and owns each part.
-
Mirroring (Microsoft Fabric)
Mirroring replicates an operational database into OneLake continuously, with no pipeline to build. It is still a copy, with lag and retention to manage.
-
Database Ingestion and Change Data Capture
Build database copies with explicit snapshot boundaries, safe progress tracking, delete handling, and reconciliation.