DataOps
Treating analytics as code, versioned and tested. The practices are borrowed from DevOps; the hard part — who owns which number — is not a tooling problem.
-
-
Transformation as Code with dbt
Shipping application code changes behavior; shipping transformation code changes history too. What that asks of models, tests, documentation, and deployment.
-
Workflow Orchestration
An orchestrator is not a fancier cron. It keeps state per run, enforces order, retries within limits — and still says nothing about whether the data is right.
-
Pipeline Orchestration and Scheduling
Running at 07:00 is not the same as running when the data is there. How orchestrators decide what runs, in what order, and what happens when a step fails.
-
Dead Letter Queue
Failed messages go somewhere instead of vanishing. That only helps if someone reads the queue — an unwatched DLQ looks exactly like data loss.
-
Delivery Semantics
Three promises about how many times a record arrives. The useful question is not which one a product advertises, but where its guarantee stops.
-
Pipeline Correctness: Duplicates and Loss
Every job succeeded and the total is still wrong. Here is where duplicates and missing rows come from, what “exactly once” really promises, and how to verify.
-
Incremental Loading and CDC
A nightly job filtering on updated_at looks right until rows quietly go missing. Here is how each change-capture method works, and what none of them can see.
-
Training-Serving Skew and Feature Stores
Simulated replays separate feature leakage, staleness, and serving-input bugs. Shared definitions help, but timing and retrieval still need verification.
-
Dual Running
During a data platform migration, both pipelines process the same inputs and a comparison reports what differs. What makes the comparison trustworthy is the shared input, the key used to match rows, and a window long enough to see rare events.
-
Data Quality Invariants and Reconciliation
Learn what exact invariants, statistical checks, and reconciliation against an independent reference can and cannot detect, with detection and false-positive arithmetic.
-
Compare-and-Swap
Check the expected version before publishing a replacement, handle conflicts deliberately, and distinguish CAS from idempotency and fencing.
-
Test Oracle
Define how a test decides correctness, preserve relevant duplicates and tolerances, and avoid deriving expected results from the same flawed logic.
-
Test Fixture
Create controlled input and environment for a test, include boundary cases, and clean up resources without losing the evidence needed to debug.
-
Fencing Token
Reject writes from obsolete worker generations at the protected resource, and distinguish an enforced fencing token from a scheduler lock or lease.
-
Incremental Processing
Update results from bounded changes, identify affected outputs, and distinguish incremental work from complete capture and safe retries.
-
ETL, ELT, and Data-Pipeline Design Patterns
ETL and ELT differ in where the transformation runs. The patterns underneath them decide whether a pipeline can be rerun without breaking anything.
-
Batch Load Ledger
Bind each batch ID to immutable input, commit its ledger entry with the load, and understand why different IDs can still duplicate the same records.
-
Data Quarantine
Retain rejected records with reasons and identity, decide when partial output is acceptable, and provide a traceable route for corrected data.
-
Data Backfill
Recompute a bounded historical range with explicit inputs and logic, account for downstream dependencies, and coordinate with ordinary loads.
-
Upsert and Merge
Use keys and source versions to insert or update rows, reject ambiguous conflicts, and distinguish upsert from delete handling and history.
-
Partition Overwrite
Replace a declared partition with its complete desired contents, handle empty partitions, and avoid using changed rows as a replacement snapshot.
-
Data Staging Area
Isolate a load candidate before publication, retain the evidence needed for replay, and distinguish a staging area from a temporary table.
-
ETL and ELT
Compare ETL and ELT by the chosen load boundary, locate each transformation, and understand what retained input makes possible after a logic change.