Data Observability

Data observability is the continuous monitoring of a dataset’s observable behavior — how much arrives, when, in what shape, and with what distribution of values — so that changes nobody wrote a rule about still become visible. It complements testing rather than replacing it: a check answers a question someone thought to ask, and observability surfaces the questions nobody asked.

The distinction is easiest to see through a failure. A table that normally receives two million rows receives two hundred thousand. Every column is the right type, no key is duplicated, nothing is null where nulls are forbidden. The suite passes, because every assertion in it was written about individual rows — and in dbt’s phrasing a test is a select statement that “seeks to grab ‘failing’ records” and passes when it returns none. No row present is wrong, so nothing fails.

That is a property of the assertions people typically write, though, not a limit on what an assertion can do. Absence is checkable whenever you can say what should have been there:

• Against an expected set. An anti-join from a dimension or a list of active sources returns the members with no rows today — missing stores, missing regions, a source that stopped.
• Against an expected count. If a daily load should produce one partition per source, assert the number of partitions rather than the contents of the ones that exist.
• Against the other side. A reconciliation against the source system’s own total fails when rows were dropped, without anyone predicting which rows.

So the real division is not “tests cannot see absence, monitoring can.” It is between checks that compare data against something you specified in advance — a rule, an expected set, a reconciliation total — and monitoring that compares today against how this dataset normally behaves. Write the dataset-level assertions where you can state the expectation; that is cheaper and more precise than an anomaly alert, and it fails with a clear reason. What observability adds is the case where you could not state it: a distribution that shifted, a volume that halved for a source nobody thought to enumerate, a change in shape that no rule anticipated.

Four things worth watching

The fourth row is the one that catches what contracts cannot. A field that changes meaning keeps its type and its name; what moves is the shape of the values.

Not the same as system observability

System observability watches the machinery: job outcomes, run durations, error rates, resource use. Data observability watches what the machinery produced. Both are necessary, and the gap between them is where the worst failures live, because the most dangerous data problems are the ones in which everything reports success.

A pipeline that finished in twelve minutes with no errors, having read an empty file, is green on every system dashboard. So is one that joined two tables and silently multiplied its row count. Neither can be detected by watching the job; both are obvious from watching the data.

What it needs, and what it gets wrong

Observability works by comparison, so it needs a baseline — history long enough to know what normal looks like for this dataset, including its weekly and seasonal shape. That requirement has consequences worth anticipating.

• New datasets have no baseline. For the first weeks, expected inventories and explicit checks carry the load; observability becomes useful later.
• Legitimate changes look like incidents. A marketing campaign, a new region, a deliberate filter change — each produces exactly the pattern an anomaly detector is built to flag. Without a way to record “this was intended,” alerts train people to ignore them.
• Detection is not diagnosis. Knowing that volume fell by ninety percent does not say which upstream stopped. Pairing the signal with lineage is what turns an alert into a starting point.

A sensible order of adoption follows from that: start with volume and per-source freshness on the datasets that matter most, because they are cheap and catch the majority of real failures. Add schema and distribution where a meaning change would be expensive. Resist the urge to instrument everything at once — an observability platform producing alerts nobody triages has the same effect as no monitoring at all, with more cost.

What happens once something is found — stopping the spread, establishing extent, correcting, telling consumers — is covered in data incident. How observability fits with tests, thresholds, and contracts is worked through in Passing Tests, Wrong Numbers.

References: GOV.UK, Meet the data quality dimensions; dbt Documentation, Data tests.