Data Quality and Data Contracts

In an illustrative scenario, a data quality suite is entirely green while the numbers it guards are wrong. Every column has the right type, no key is duplicated, nothing is null where nulls are forbidden — and revenue is overstated by twelve percent because a refund flag changed meaning three weeks ago. This is not a failure of testing. It is the boundary of what testing does, and knowing where that boundary falls is most of what makes a quality program useful rather than reassuring.

Six dimensions, and what each can tell you

Data quality is usually broken into dimensions, and the UK government’s published definitions are as good as any: accuracy is that “data reflects reality”; completeness that “all the data required for a particular use is present and available to be used”; uniqueness “measures the number of duplicates”; consistency that “data values do not conflict with other values within a record or across different data sets”; timeliness is “whether the data is available when expected and needed”; and validity is “the extent to which the data conforms to the expected format, type, and range.”

Listed that way they look equivalent. They are not, and the difference that matters operationally is how much of each a machine can confirm on its own.

Dimension	How far an automated check gets	What still needs a person
Validity	All the way. Format, type, and range are decidable	Whether the expected range is still the right one
Uniqueness	All the way, given a key	Which key defines “the same thing” — a business decision about grain
Consistency	Within and across known datasets, where the rule is stated	Which disagreements are errors and which are legitimate differences in context
Timeliness	Measurable against a stated deadline — see freshness	What deadline the business actually needs
Completeness	Checkable, but only against an independent expectation. A row-level rule reads only the rows that arrived; a dataset-level assertion can compare what is present with what should be — an anti-join against expected keys, or a count against an expected number of partitions	Establishing what should have arrived, independently of what did
Accuracy	Almost nowhere by itself. “Reflects reality” needs a reference outside the data	Comparison against a source of truth, and judgment about the remaining difference

The bottom two rows are where most quality programs quietly stop. They are also where the expensive failures live.

Technically valid, business wrong

The distinction worth installing in a team’s vocabulary is between technical validity — the data is well formed under the rules we wrote — and business plausibility — the data describes what happened. They fail independently, and only the first has a natural owner in engineering.

A few examples make the gap concrete. An order amount of 4,500,000 passes a positive-number check and is a decimal error. A country code of GB is valid and wrong if the source switched to UK for a subset of records. A daily total that arrived on time, with every field populated, is missing a third of stores because one upstream export silently stopped. Each one is invisible to the checks a pipeline usually carries, and obvious to someone who knows the business.

Two mechanisms close most of that gap, and both need something the output alone cannot supply. The first is reconciliation against the source: an independent comparison of counts, totals, or key ranges at an agreed cutoff — the way to notice records that never arrived when the only reference is the source itself. The second is a stated expected inventory — delivery completeness defined so that a source sending nothing produces a trigger rather than silence.

What a check actually is, and what severity means

Mechanically, most quality checks take one form. dbt’s documentation describes it plainly: data tests “are select statements that seek to grab ‘failing’ records, ones that disprove your assertion,” and “if the data test returns zero failing rows, it passes, and your assertion has been validated.” Tests come in two shapes — a singular test, which is a saved query returning failing rows, and a generic test, “a parameterized query that accepts arguments” that can be referenced by name on models, columns, sources, and more.

Because the form is simple, the interesting decision is not what to check but what a failure should do. dbt exposes this as severity: a test can be set to warn or error, and “you can use the error_if and warn_if configs to set custom failure thresholds in your tests.” That maps onto a decision the organization has to make anyway.

• Blocking. Consumers keep the previous good version rather than seeing the failed one. Right when a wrong number is worse than a stale one — most financial and regulatory reporting.
• Warning. The data is published with the failure recorded and visible. Right when a stale number is worse than an imperfect one — most operational dashboards.

Blocking is a publishing structure, not a severity setting. Setting a test to error does not restore anything. A test usually runs after the model it checks has already been built, so the table now holds the bad data; what the failure does is stop the tasks downstream of it. Keeping the previous version visible requires the pipeline to be built for it: write the new data to a staging or versioned relation, run the checks against that, and only then move the view or alias that consumers read. Without that separation, “blocking” means consumers see the failed data while the rest of the run is skipped — which is close to the opposite of the intent.

Deciding this per rule, rather than setting everything to one or the other, is what keeps a suite from becoming a wall nobody can ship past or an alert stream nobody reads. And the numbers in those rules deserve their own justification: a quality threshold should be derived from the decision it protects, not chosen round.

What a contract can promise

A data contract is where these promises stop being folklore. It states what a producer supplies and what consumers may rely on — and its useful property is that it covers meaning as well as structure: one row per order, an amount in the named currency’s minor units, a stated time basis, a named owner, an expected freshness, and what happens when a check fails.

A contract can state quality, delivery, and support obligations. Writing them down or passing a structural check does not ensure they have been fulfilled; each obligation needs measurement and an enforcement or response process.

What a contract can specify	What still needs verification or enforcement
Structure, semantic definitions, and accuracy targets, with compatibility rules for change	A conforming row may still be wrong; verify semantic and accuracy claims against suitable evidence
Freshness, completeness, and delivery obligations at a stated measurement boundary	Observe actual delivery and define escalation and recovery when the source misses its obligation
A named owner, support hours, and incident response commitments	Ensure coverage and capacity exist; a written 03:00 response commitment does not staff itself
Notice of structural or semantic changes and a migration path	Review meaning changes and their effects; unchanged columns alone do not reveal a changed definition of paid

A semantic change that preserves the schema can pass structural checks. Other semantic tests, reconciliations, or review may detect it. Keep the metric definition versioned and reviewed with the contract, and make clear which obligations each check verifies.

Watching versus checking

Checks express expectations that can be evaluated; data observability also helps investigate current state and changes in volume, freshness, schema, and distribution. A fall from two million rows to two hundred thousand may warrant investigation even when row-level tests pass. It is not automatically an error: a planned filter or lower activity can produce the same change. Dataset-level assertions can also catch the drop when an expected inventory or count is defined.

Execution health and data quality are different observation targets, but they can share tooling. A job that reads an empty file may pass an exit-status check while failing an expected-input check. Expose the relevant counts, timestamps, and check results through the monitoring system instead of assuming that every successful run produced acceptable data.

When it fails: who does what, in what order

Bad data that has reached consumers is closer to an incident than a bug, and the order of operations matters more than the repair. A data incident runs roughly like this: contain the affected paths by withholding publication, quarantining output, or pausing selected dependent jobs; at the same time, warn consumers about the suspected impact and whether to stop using figures they already hold. Establish which periods, keys, tables, and reports are affected, and update that warning as the investigation proceeds. Fix and verify the affected results, then communicate the confirmed corrections and when use can resume. Notification cannot wait for reprocessing to finish: stopping a job does not recall a downloaded file.

Rejected records belong somewhere they can be examined rather than dropped. Data quarantine preserves that evidence, but an error that passed the checks may never have been quarantined. Retained source snapshots, published versions, transformation history, lineage, and reconciliation can help reconstruct its extent. Late data and lookback defines how later arrivals change results; incident response addresses broken promises and consumer impact. Arrivals handled within the agreed lateness allowance can be normal operation. A missed completeness or freshness commitment, or an incorrect figure already distributed, may require incident response and correction notices alongside the technical update.

Responsibility follows the roles rather than the tooling. The owner decides whether a figure is withheld or published with a caveat; the steward establishes what happened and what the corrected definition should be; the custodian runs the repair. Where those three are the same person, the decisions still have to be made in that order — the common failure is skipping straight to the repair and discovering afterwards that nobody agreed what the right answer was.

Reporting quality without lying about it

A single quality percentage is the most common executive artifact and the least informative. Ninety-seven percent of what, measured how, over which period, with which checks — and how many of those checks exist at all? A number that rises because someone deleted a failing test is indistinguishable from one that rises because the data improved.

Report coverage alongside outcomes: the share of critical datasets that have a contract and an owner, failures by severity and time to resolution, and the results of reconciliation against sources — which approaches accuracy questions without itself measuring accuracy. Reconciliation answers whether the destination holds what the named source holds; faithfully copying a value the source recorded wrongly passes it. Accuracy in the sense of “data reflects reality” needs something outside the pipeline: an independent source, a business artifact the value can be checked against, or a sampled manual verification. Stated as targets someone owns — the discipline an objective needs — they turn quality from a status color into a commitment that can be met or missed.

References

• GOV.UK, Meet the data quality dimensions
• dbt Documentation, Data tests