The Catalog Nobody Opens: Metadata, Lineage, and What They Are Actually For

A catalog bought for discovery often ends the same way. A scan runs, thousands of tables appear with their columns and types, and six months later the people who were supposed to use it are still asking a colleague which table to query. The tool worked exactly as advertised. It collected the metadata that can be collected automatically, which is not the metadata that answers the question those people arrived with.

Three kinds of metadata, three different sources

Kind	What it is	Where it comes from
Technical	Tables, columns, types, partitions, sizes, physical locations	Scanning — cheap enough that every catalog is full of it. “Complete” only within what the scanner can reach: it needs credentials, a supported connector, and a recent run, so coverage and freshness are properties of the collection setup rather than guarantees
Business	What a field means, which definition a number uses, who is accountable, what consumers may rely on	People deciding it. A tool cannot invent the definition — but once it is declared somewhere machine-readable, such as a model or source file in the transformation project, collection can propagate it rather than asking everyone to retype it in a catalog
Operational	When a dataset was last loaded, how many rows arrived, which job produced it and whether it succeeded, who queried it	Pipelines and query engines emitting records as they run — see operational metadata

The imbalance is one reason a catalog stays empty of what people want, and the easiest to check: compare the share of assets with a description or an owner against the share with a scanned schema. Technical metadata arrives with the scanner; business metadata requires a decision by someone who has other work; operational metadata requires instrumentation nobody asked for until the first incident. A catalog left to fill itself will therefore contain the kind that answers engines’ questions — where a table is, how it is partitioned — and few of the ones people bring to it.

Which is why catalog coverage — the share of assets carrying an owner, a description, and a classification — is the measure that matters rather than the number of registered tables. A tool can register everything and cover nothing.

What lineage is for

Data lineage records where data came from and what happened to it on the way, and this article concentrates on the two uses that arise most often around change and incidents — worth stating because lineage diagrams are frequently built for no question at all. Other uses are real too: tracing where sensitive data has propagated, and proving where a figure came from.

• Forward, before a change. This column is being removed — what breaks? That is impact analysis, and it is the difference between announcing a change and discovering its consequences from a complaint.
• Backward, during an incident. This figure is wrong — which upstream input could have caused it? Lineage narrows a search that would otherwise start with guessing.

Each use is a question someone is asking. A lineage graph built to be looked at, rather than to answer a question, becomes a picture people screenshot for slides.

Granularity decides whether it helps

Table-level lineage says that the orders table feeds the revenue report. That is enough to start an investigation and not enough to finish one: if the report depends on four of forty columns, a change to one of the other thirty-six does not affect it — provided the graph captured every way the report uses the table, including columns it filters on without selecting — and table-level lineage cannot tell you which case you are in. Column-level lineage narrows the blast radius from “everything downstream” to “these three dashboards,” which is the difference between a change being a conversation and being a project.

The cost is that column-level lineage has to be derived from the transformation logic rather than from the fact that one job read one table. The condition is not whether the code looks static: it is whether the collector can see the query that actually ran, the plan it produced, or an explicit mapping the job reports itself. Code that builds SQL at run time still yields column edges if the final statement or its plan is captured; a perfectly static model yields none if no collector covers that engine. A job that copies a file opaquely is the genuinely hard case.

One more requirement decides whether the graph can answer the question people bring to it. “Is it safe to change this column” needs the indirect dependencies too — a field used in a join or filter shapes the output without appearing in it. A graph that records only columns visible in the SELECT list will call such a change harmless.

Where lineage comes from

The reliable way to collect lineage is to take it from actual executions — emitted by the systems that process data as they run, or read afterwards from the artifacts and logs a run left behind — rather than to infer it from static code or written documentation. OpenLineage’s dbt integration, for example, supports both: reading a run’s JSON artifacts after the fact and consuming its structured logs while it runs. OpenLineage exists to standardize exactly that: it describes itself as “an open framework for data lineage collection and analysis” and “an Open Standard for lineage metadata collection designed to record metadata for a job in execution.”

Its model is worth knowing because it is the shape most tools converge on — dataset, job, and run entities “uniquely identified using consistent naming strategies,” with integrations “pushed to the underlying scheduler and/or data processing framework,” and facets as “user-defined metadata” that enriches those entities. Two consequences follow for anyone planning this work. Lineage taken from a run reflects what actually happened rather than what the code appears to do, which matters when a job is configured differently in production — though having been emitted during a run does not by itself mean every path was observed or that the data change it describes was committed; the run identifier, its status, and the collector’s scope have to be kept with it. And consistent naming across systems is the hard part: the same table referred to three ways by three tools produces three disconnected graphs.

Why the catalog goes unused

Four reasons, any of which may be the real one; check each before assuming.

1. It answers the wrong question. People arrive asking “which table should I use for revenue this quarter” and the catalog offers a search over table names. The answer they need is a curated recommendation with a metric definition attached, not a directory.
2. Nobody was asked to fill it in. Business metadata is work, and work that is expected but not assigned does not happen. If domains are meant to populate descriptions, that expectation has to be stated, made easy, and checked — which is what coverage measures.
3. It is not where the work happens. A catalog that requires opening a separate tool competes with asking a colleague in a chat window. It loses when the colleague is faster or more trusted, and wins when it finds the right asset with less effort and with permissions and freshness the colleague cannot vouch for. Where usage logs show people never open it, surfacing definitions and ownership inside the query editor and the BI tool can matter more than adding features inside the catalog; whether it does depends on where the users’ bottleneck actually is.
4. It is not trusted. One stale description teaches a reader that descriptions may be stale, and a catalog people verify elsewhere is a catalog they stop opening. Fewer entries that are known-current beat complete entries of unknown age — which argues for showing when each description was last confirmed, and by whom.

Read together, they point at the same remedy: a catalog meant for people succeeds as the place where a small number of important decisions are recorded and reachable, and a complete inventory of everything that exists does not by itself get it used. The inventory has jobs of its own — engines resolving schemas and locations, access, retention, and cost management — and it contributes to discovery too: names, columns, types, and locations are how a search starts. What it cannot settle on its own is whether the asset found is the right one for the purpose, which needs the definition, the owner, and the intended use.

What to do first

Pick one question the organization asks repeatedly and cannot answer — “which dashboards break if this column changes” is the usual candidate — and make the metadata that answers it complete for the twenty datasets that matter most. That means an owner per dataset, a definition where a number is involved, a contract where consumers depend on it, and lineage at whatever granularity the engines can emit.

Two supplements are worth having from the start. Query logs show which accounts actually read a table in the last ninety days, including consumers no catalog knows about, and they are usually already being collected — with one step left: a shared BI account or an application’s connection pool names a principal, not the dashboard or the person, so a query tag, the BI tool’s metadata, or a conversation still has to close that gap — often in the same audit log kept for compliance. And a business glossary is where the agreed meaning of a term lives once, rather than being restated differently in every dataset description.

Then measure the thing that is supposed to improve: how long it takes to answer the question you chose, before and after. Registered-asset counts rise regardless of whether anything got better; the time to answer a real question does not.

References: OpenLineage Documentation; OpenLineage, dbt integration; Snowflake Documentation, QUERY_HISTORY view. Checked September 2026.