Data Interval
The 07:00 run usually processes yesterday, not this morning. Naming the interval a run covers is what makes a rerun mean the same thing as the original.
-
-
Pipeline Correctness: Duplicates and Loss
Every job succeeded and the total is still wrong. Here is where duplicates and missing rows come from, what “exactly once” really promises, and how to verify.
-
Incremental Snapshot
Seeding a table no longer has to mean one long locked scan. Chunks and watermarks let a snapshot run beside the change stream, and pause and resume.
-
Production Airflow: Idempotency and Backfills
Reproduce the failures that corrupt data in scheduled pipelines, from duplicate writes after a lost acknowledgement to a backfill that overwrites a correction, and fix each one.
-
Airflow Operations: Retries and Backfills
Follow one failed run through logical time, retries, catchup, backfill, worker and scheduler failure, and see which rules keep a repeated run from corrupting data.
-
Building Incremental and Idempotent Pipelines
Define incremental bounds and retry identity, preserve newer updates and deletes, and coordinate publication, progress, and backfills.
-
ETL, ELT, and Data-Pipeline Design Patterns
ETL and ELT differ in where the transformation runs. The patterns underneath them decide whether a pipeline can be rerun without breaking anything.