Asset-Oriented Orchestration: Dagster and Prefect
Declare data assets instead of tasks, record what each production run created, map partitions to find what a correction affects, and gate publication with checks.
-
-
Production Airflow: Idempotency and Backfills
Reproduce the failures that corrupt data in scheduled pipelines, from duplicate writes after a lost acknowledgement to a backfill that overwrites a correction, and fix each one.
-
Data Pipeline CI/CD and Environment Promotion
A code rollback does not undo data a bad release already wrote. See how fixtures, contract tests, isolated builds, and staged promotion limit that damage, and how to repair it.
-
Airflow Operations: Retries and Backfills
Follow one failed run through logical time, retries, catchup, backfill, worker and scheduler failure, and see which rules keep a repeated run from corrupting data.
-
Transformation as Code: Models and Tests
Trace how a SELECT statement becomes a model in a dependency graph, how data tests and unit tests differ, and why an incremental model can drift from a full rebuild.