Survivorship Rule
Two systems hold different addresses for one customer, both updated this month. Deciding which one the merged record shows is not a technical question.
-
-
Entity Resolution
Matching is a judgment under uncertainty. Merging two different customers and splitting one across two records are both errors, and only one is easy to undo.
-
Master Data Management
Five systems each hold a customer record and none of them is wrong. MDM is the decision about what to do with that, not a product that removes it.
-
Quality Threshold
A threshold needs a defined population, window, and acceptable impact. Derive its value from the decision it protects rather than choosing a percentage without a basis.
-
Data Incident
Bad data in production is closer to an incident than a bug. While the affected data keeps being published, copied, or read, the decisions and outputs needing correction can grow.
-
Data Observability
A large change in row count calls for investigation. Expected-count tests, reconciliation, current-state observation, and anomaly detection provide complementary evidence.
-
Data Quality and Data Contracts
A green test suite means the data broke none of the rules you thought to write. Whether the numbers are correct is a different question with a different owner.
-
DataOps
Treating analytics as code, versioned and tested. The practices are borrowed from DevOps; the hard part — who owns which number — is not a tooling problem.
-
Source Freshness Check
A build on stale input succeeds and looks fine. A freshness check compares the newest loaded row against a threshold, so the silence becomes an alert.
-
Transformation as Code with dbt
Shipping application code changes behavior; shipping transformation code changes history too. What that asks of models, tests, documentation, and deployment.
-
Pipeline Correctness: Duplicates and Loss
Every job succeeded and the total is still wrong. Here is where duplicates and missing rows come from, what “exactly once” really promises, and how to verify.
-
When Correct SQL Returns the Wrong Answer
A query can run cleanly and still answer a different question. See how joins, NULLs, ordering, and aggregation levels change results, and how to catch it.
-
Schema-on-Read
Schema-on-read stores data as it arrives and decides its structure at query time. See what that flexibility buys, what it pushes downstream, and when to use it.
-
Outlier
With one contaminating value the largest z-score was 17.8407. With thirty of them it was 4.0420 — they inflate the deviation they are measured against.
-
Full Annotation
Leaving 80% of training positives unlabeled reduced recall from 0.6341 to 0.0031 in this experiment. Evaluation-label omissions have different effects.
-
Label Noise
Random label noise in training barely moved accuracy, 0.7732 to 0.7681. The same noise in the test set took it to 0.5595, and AUROC to 0.5743.
-
Accepted Break
Not every difference found in a data migration is worth fixing. An accepted break records the ones the team will live with: who approved them, why, how far they extend, and when the approval lapses.
-
Dual Running
During a data platform migration, both pipelines process the same inputs and a comparison reports what differs. What makes the comparison trustworthy is the shared input, the key used to match rows, and a window long enough to see rare events.
-
Data SLIs, SLOs, and Incident Response
A green job does not mean a reliable dataset. Learn to define data indicators, set objectives, alert on budget burn, find silent failures, and tell consumers what to do.
-
Master Data and Entity Resolution Pipelines
Follow records from several systems through blocking, scoring, clustering, and survivorship, and see how thresholds, chaining, and corrections decide merge quality.
-
Data Quality Invariants and Reconciliation
Learn what exact invariants, statistical checks, and reconciliation against an independent reference can and cannot detect, with detection and false-positive arithmetic.
-
Semantic Layers and Consistent Metric Definitions
One question, “what was March revenue?”, can honestly return 600, 550, or 100 from the same seven orders. This post shows how metric contracts, aggregation rules, time-window rules, versions, and consumer tests make one answer.
-
Testing and Debugging Spark Data Pipelines
Build fixtures and independent expected results, test real storage boundaries, and reduce Spark failures into useful regression tests.
-
Edge Case
Learn what an edge case is, why boundary rows break rules that work on typical data, and how to decide the behavior instead of discovering it.
-
Data Quality
Understand six data quality dimensions and why passing one check does not establish fitness for every use.