Quality Threshold

A quality threshold is the point at which a data check stops passing: zero null customer identifiers, no more than 0.1 percent of rows failing a range rule, freshness no worse than two hours. Tools expose it directly — dbt lets a test’s severity be warn or error and provides error_if and warn_if configs “to set custom failure thresholds in your tests,” and source freshness takes warn_after and error_after as a count and a period.

Setting the number is the part that gets done badly, because a threshold looks like a technical parameter and is actually a statement about what the organization is willing to publish.

Derive it from what it protects

The reliable method is to work backwards from the decision that consumes the data, which usually produces a number nobody would have chosen by feel.

• A deadline gives a freshness threshold. With no other work or delay, a report due at 08:00 after a 40-minute build needs its input by 07:20. Choosing a further 20 minutes for variation or retries moves the operating deadline to 07:00. The first time is the arithmetic limit; the second includes an explicit margin.
• A materiality rule gives a tolerance. Display precision does not determine an acceptable error. Rounded to the nearest thousand, 10,490 becomes 10,000 and 10,510 becomes 11,000: a difference of 20 crosses the display boundary. Set an approved tolerance for the use, considering rounding boundaries, accumulated errors, and business impact. An unchanged display is not evidence that an error is acceptable for a payment or another use.
• An identity rule gives a zero. For a contractually required primary key, duplicates or nulls violate the identity guarantee. A nullable foreign key representing an optional relationship is different: missing values can be valid if the join and consumer contract define how to handle them. Specify the column, its role, and the affected use before setting a zero tolerance.

That last category is worth separating explicitly. A required identity invariant must hold for the dataset being published; a global allowance for bad rows cannot substitute for it. Other rules permit missing or uncertain values under a stated consumer contract. A tolerable count must still be assessed against which records and decisions are affected. Mixing the two produces suites where a genuine corruption passes because it fell under a percentage someone set globally.

Two levels, not one

A single threshold forces one response to every degree of failure. Two — a warning level and a blocking level — let the check say something useful at both. For a hypothetical daily customer profile used only for a noncritical display, an owner might approve a warning at 1 percent missing optional nicknames and a block at 20 percent. Those values are examples, not universal limits or proof that the load is broken. Missing payment identifiers would need a different rule. Even a low row rate may warrant a hold if it affects high-value amounts, critical users, or an impact that has not been bounded. A warning or error threshold classifies the check result; it does not preserve a previous version. Keeping consumers on validated data requires a separate publication design: build in staging or a versioned destination, validate, then switch the consumer-visible version. A check run after overwriting the public table cannot provide that protection.

Both levels need the same four elements any measurable target needs: the population the rule applies to, where it is measured, over what window, and the value that counts as failing. A threshold without a window — “no more than 1 percent nulls” — is not enforceable until someone says whether that is per load, per day, or per month.

The failure mode to watch for

Thresholds drift downward under delivery pressure. A check fails, the release is urgent, the threshold is relaxed “temporarily,” and the suite returns to green without anything improving. This is observable and worth observing: a record of threshold changes, with who changed each one and why, distinguishes a deliberate recalibration from an accumulation of exceptions.

Two habits keep it honest. Give every threshold an owner — the person whose decision it protects, not the engineer who wrote the test — and a review date, because a number derived from a deadline stops being right when the deadline moves. And when a threshold is loosened to let something through, record it as an accepted risk with an expiry rather than as a settled value. Stated as a target someone owns, the whole arrangement is the same discipline an objective needs; without an owner it is a preference that erodes.

How thresholds fit with checks, contracts, and incident response is worked through in Passing Tests, Wrong Numbers.

References:

• dbt Documentation, Data tests
• dbt Documentation, severity, error_if, and warn_if
• dbt Documentation, Sources (source data freshness)