Data Observability

Data observability is the continuous monitoring of a dataset’s observable behavior — how much arrives, when, in what shape, and with what distribution of values — so that changes nobody wrote a rule about still become visible. It complements testing rather than replacing it: a check answers a question someone thought to ask, and observability surfaces the questions nobody asked.

The distinction is easiest to see through a failure. A table that normally receives two million rows receives two hundred thousand. Every column is the right type, no key is duplicated, nothing is null where nulls are forbidden. The suite passes, because every assertion in it was written about individual rows. dbt states the mechanism plainly: “Data tests return a set of failing records.” And: “If the data test returns zero failing rows, it passes, and your assertion has been validated.” A row that is not there cannot be selected, so nothing fails.

That is a property of the assertions people typically write, though, not a limit on what an assertion can do. Absence is checkable whenever you can say what should have been there:

• Against an expected set. An anti-join from a dimension or a list of active sources returns the members with no rows today — missing stores, missing regions, a source that stopped.
• Against an expected count. If a daily load should produce one partition per source, assert the number of partitions rather than the contents of the ones that exist.
• Against the other side. A reconciliation against the source system’s own total fails when rows were dropped, without anyone predicting which rows.

So the real division is not tests cannot see absence, monitoring can. Three kinds of check overlap here, and each sees something the others miss: assertions against something you specified in advance — a rule, an expected set, a reconciliation total; observation of the current state, which needs no history at all — what schema actually arrived, when the last row landed, which step failed, what the lineage says depends on it; and anomaly detection, which compares today against how this dataset normally behaves and is the only one of the three that requires a baseline. Write the dataset-level assertions where you can state the expectation; that is cheaper and more precise than an anomaly alert, and it fails with a clear reason. What the other two add is the case where you could not state the expectation: a distribution that shifted, a volume that halved for a source nobody thought to enumerate, a change in shape that no rule anticipated.

Four things worth watching

The fourth row is the one that catches meaning changes a type check would miss: a field that changes meaning keeps its type and its name, and what moves is the shape of the values. Neither half of that is a guarantee. A contract can carry more than types — a declared set of permitted categories, a stated unit, a currency code, a foreign key into reference data — and those catch a class of meaning change at the boundary, which is cheaper than inferring it later. And a meaning change need not disturb the distribution at all: if a status code keeps the same frequencies while its definition changes, or amounts switch from one currency to another of similar magnitude, nothing looks anomalous. Those are found by reading the change — an upstream release note, a schema review, a question to the producing team — which is why the producer relationship is part of the control and not an alternative to it.

Not the same as system observability

Execution health and output quality answer different questions. Job outcomes, durations, errors, and resource use describe execution; record counts, freshness, and content checks describe what it produced. The same telemetry pipeline and dashboard can carry both. A successful exit status alone does not establish that consumers received correct data.

A job can finish in twelve minutes after reading an empty file, or complete a join that multiplies rows. An exit-status and duration dashboard may miss either problem. Instrumented input counts, output counts, and quality-check results can reveal them in the job’s own monitoring, provided expectations distinguish a failure from a legitimate workload change.

What it needs, and what it gets wrong

The anomaly-detection part works by comparison, so it needs a baseline — history long enough to know what normal looks like for this dataset, including its weekly and seasonal shape. The state-observing part does not: schema, arrival times, failure locations and lineage can be read on day one. That split decides what the first weeks look like.

• New datasets have no baseline. What is unavailable at first is anomaly detection, not observation: from the first load you can see the schema that arrived, whether every expected partition landed, how old the newest record is and where a run failed. Expected inventories and explicit assertions carry what needs stating in advance, and the comparison against history joins later.
• Legitimate changes look like incidents. A marketing campaign, a new region, a deliberate filter change — each produces exactly the pattern an anomaly detector is built to flag. Without a way to record that a change was intended, alerts train people to ignore them.
• Detection is not diagnosis. Knowing that volume fell by ninety percent does not say which upstream stopped. Pairing the signal with lineage is what turns an alert into a starting point.

Start with measurements that protect the most important dataset commitments. Volume and per-source freshness are often practical candidates, with schema and distribution checks where those changes matter. Evaluate coverage against known failures rather than assuming these signals catch a fixed majority. Assign ownership and response rules so observations lead to investigation or action.

What happens once something is found — stopping the spread, establishing extent, correcting, telling consumers — is covered in data incident. How observability fits with tests, thresholds, and contracts is worked through in Passing Tests, Wrong Numbers.

References

• GOV.UK, Meet the data quality dimensions
• dbt Documentation, Data tests