Passing Tests, Wrong Numbers: Data Quality and What a Contract Can Promise

A data quality suite can be entirely green while the numbers it guards are wrong. Every column has the right type, no key is duplicated, nothing is null where nulls are forbidden — and revenue is overstated by twelve percent because a refund flag changed meaning three weeks ago. This is not a failure of testing. It is the boundary of what testing does, and knowing where that boundary falls is most of what makes a quality program useful rather than reassuring.

Six dimensions, and what each can tell you

Data quality is usually broken into dimensions, and the UK government’s published definitions are as good as any: accuracy is that “data reflects reality”; completeness that “all the data required for a particular use is present and available to be used”; uniqueness “measures the number of duplicates”; consistency that “data values do not conflict with other values within a record or across different data sets”; timeliness is “whether the data is available when expected and needed”; and validity is “the extent to which the data conforms to the expected format, type, and range.”

Listed that way they look equivalent. They are not, and the difference that matters operationally is how much of each a machine can confirm on its own.

Dimension	How far an automated check gets	What still needs a person
Validity	All the way. Format, type, and range are decidable	Whether the expected range is still the right one
Uniqueness	All the way, given a key	Which key defines “the same thing” — a business decision about grain
Consistency	Within and across known datasets, where the rule is stated	Which disagreements are errors and which are legitimate differences in context
Timeliness	Measurable against a stated deadline — see freshness	What deadline the business actually needs
Completeness	Checkable, but only against an independent expectation. A row-level rule reads only the rows that arrived; a dataset-level assertion can compare what is present with what should be — an anti-join against expected keys, or a count against an expected number of partitions	Establishing what should have arrived, independently of what did
Accuracy	Almost nowhere by itself. “Reflects reality” needs a reference outside the data	Comparison against a source of truth, and judgment about the remaining difference

The bottom two rows are where most quality programs quietly stop. They are also where the expensive failures live.

Technically valid, business wrong

The distinction worth installing in a team’s vocabulary is between technical validity — the data is well formed under the rules we wrote — and business plausibility — the data describes what happened. They fail independently, and only the first has a natural owner in engineering.

A few examples make the gap concrete. An order amount of 4,500,000 passes a positive-number check and is a decimal error. A country code of GB is valid and wrong if the source switched to UK for a subset of records. A daily total that arrived on time, with every field populated, is missing a third of stores because one upstream export silently stopped. Each one is invisible to the checks a pipeline usually carries, and obvious to someone who knows the business.

Two mechanisms close most of that gap, and both need something the output alone cannot supply. The first is reconciliation against the source: an independent comparison of counts, totals, or key ranges at an agreed cutoff — the way to notice records that never arrived when the only reference is the source itself. The second is a stated expected inventory — delivery completeness defined so that a source sending nothing produces a trigger rather than silence.

What a check actually is, and what severity means

Mechanically, most quality checks take one form. dbt’s documentation describes it plainly: data tests “are select statements that seek to grab ‘failing’ records, ones that disprove your assertion,” and “if the data test returns zero failing rows, it passes, and your assertion has been validated.” Tests come in two shapes — a singular test, which is a saved query returning failing rows, and a generic test, “a parameterized query that accepts arguments” that can be referenced by name on models, columns, sources, and more.

Because the form is simple, the interesting decision is not what to check but what a failure should do. dbt exposes this as severity: a test can be set to warn or error, and “you can use the error_if and warn_if configs to set custom failure thresholds in your tests.” That maps onto a decision the organization has to make anyway.

• Blocking. Consumers keep the previous good version rather than seeing the failed one. Right when a wrong number is worse than a stale one — most financial and regulatory reporting.
• Warning. The data is published with the failure recorded and visible. Right when a stale number is worse than an imperfect one — most operational dashboards.

Blocking is a publishing structure, not a severity setting. Setting a test to error does not restore anything. A test usually runs after the model it checks has already been built, so the table now holds the bad data; what the failure does is stop the tasks downstream of it. Keeping the previous version visible requires the pipeline to be built for it: write the new data to a staging or versioned relation, run the checks against that, and only then move the view or alias that consumers read. Without that separation, “blocking” means consumers see the failed data while the rest of the run is skipped — which is close to the opposite of the intent.

Deciding this per rule, rather than setting everything to one or the other, is what keeps a suite from becoming a wall nobody can ship past or an alert stream nobody reads. And the numbers in those rules deserve their own justification: a quality threshold should be derived from the decision it protects, not chosen round.

What a contract can promise

A data contract is where these promises stop being folklore. It states what a producer supplies and what consumers may rely on — and its useful property is that it covers meaning as well as structure: one row per order, an amount in the named currency’s minor units, a stated time basis, a named owner, an expected freshness, and what happens when a check fails.

Contracts are also where the limits should be written down honestly.

A contract can promise	A contract cannot promise
Structure: columns, types, keys — enforceable automatically, with compatibility rules for change	That the values are correct. A conforming row can still be wrong
Freshness and completeness targets, stated with a measurement boundary	That the source will send what it was supposed to send
A named owner and a response when a check fails	That someone will be available at 03:00 unless that was agreed separately
Notice before a breaking change, and a migration path	That a meaning change will be noticed — nothing in the structure moves when “paid” starts excluding partial refunds

That last row is the important one. Structural compatibility is a floor, not a ceiling: a change that alters what a field means passes every schema rule, which is why the meaning belongs in a written metric definition that is versioned and reviewed alongside the contract.

Watching versus checking

Checks answer questions you thought to ask. Data observability is the complement: continuous monitoring of a dataset’s behavior — volume, freshness, schema, distribution — so that a change nobody wrote a rule about still surfaces. A table that normally receives two million rows and today received two hundred thousand has broken no rule, and something is wrong.

It is worth keeping this separate from system observability, which watches the machinery: job success, latency, error rates, resource use. The two answer different questions and both are needed, because the most dangerous data failures are the ones where the machinery reports success. A pipeline that completed in twelve minutes with no errors, having read an empty file, will look healthy on every system dashboard.

When it fails: who does what, in what order

Bad data that has reached consumers is closer to an incident than a bug, and the order of operations matters more than the repair. A data incident runs roughly like this: stop the spread first by pausing dependent jobs, because every further run widens what has to be corrected; establish the extent — which periods, which keys, which downstream tables — since a repair that cannot state its scope cannot be verified; then fix and reprocess; then tell the consumers who already received the wrong numbers.

Rejected records belong somewhere they can be examined rather than dropped — data quarantine is what makes the second step possible — and records arriving after a window closed are a separate question covered by late data and lookback rather than by the incident process.

Responsibility follows the roles rather than the tooling. The owner decides whether a figure is withheld or published with a caveat; the steward establishes what happened and what the corrected definition should be; the custodian runs the repair. Where those three are the same person, the decisions still have to be made in that order — the common failure is skipping straight to the repair and discovering afterwards that nobody agreed what the right answer was.

Reporting quality without lying about it

A single quality percentage is the most common executive artifact and the least informative. Ninety-seven percent of what, measured how, over which period, with which checks — and how many of those checks exist at all? A number that rises because someone deleted a failing test is indistinguishable from one that rises because the data improved.

Three things reported together say more than any composite. The share of critical datasets that have a contract and an owner at all. The number of failures by severity, with how long each took to resolve. And the results of reconciliation against sources — which is the closest of the three to accuracy without being a measure of it. Reconciliation answers whether the destination holds what the named source holds; faithfully copying a value the source recorded wrongly passes it. Accuracy in the sense of “data reflects reality” needs something outside the pipeline: an independent source, a business artefact the value can be checked against, or a sampled manual verification. Stated as targets someone owns — the discipline an objective needs — they turn quality from a status color into a commitment that can be met or missed.

References: GOV.UK, Meet the data quality dimensions; dbt Documentation, Data tests. Checked September 2026.