Operational Metadata

Operational metadata is the record of what actually happened to data: when a dataset was last loaded, how many rows arrived, which job produced it and whether that run succeeded, how long it took, and who queried the result afterward. It sits alongside technical metadata, which describes shape, and business metadata, which describes meaning.

The distinguishing property is that it is about events rather than definitions. A column’s type is a fact about the table; a load that finished at 06:12 with 1.8 million rows is a fact about one moment, and tomorrow there will be another one. That makes it the kind that answers questions about recent events and observed state — what loaded, whether it succeeded, who used it — where technical metadata answers what the current schema is and business metadata who owns a definition now.

What it answers

• Is this current? Freshness is computed from it — dbt’s source freshness, for instance, compares the maximum of a load timestamp column with the current time.
• Did everything arrive? Row counts per source and per partition are what make an expected inventory checkable rather than aspirational.
• Who depends on this? Query logs show which accounts read a table in the last ninety days — including consumers no catalog knows about, which is usually where the surprises are; a shared service account still has to be resolved to a report or a person through a query tag, BI metadata, or asking.
• What did this cost? Compute per job and per query is operational metadata too, and it is what makes cost attributable to a workload rather than to a platform.

Each of those is a question people ask constantly and answer badly, usually by asking a person who remembers.

It comes from runs, not from scans

Technical metadata can be harvested by inspecting a system at rest. Operational metadata comes from work happening — emitted while a run executes, or read afterward from what the run left behind: logs, run artifacts, status records — which is why standards for it are built around execution rather than around scanning. OpenLineage describes itself as “an Open Standard for lineage metadata collection designed to record metadata for a job in execution,” with a model of dataset, job, and run entities “uniquely identified using consistent naming strategies” and integrations “pushed to the underlying scheduler and/or data processing framework.”

The run entity is the part worth noticing. Keeping a record per execution — not just the latest state of a table — is what lets someone reconstruct a past day during an investigation: which run produced the numbers a report used, what it read, whether it succeeded, and how its row count compared with the run before. A system that stores only the current state can say what is true now and nothing about what was true on the Tuesday in question.

Collection is automatic; keeping it useful is not

Most platforms emit this data already, which creates the illusion that nothing needs designing. Three decisions still do.

• Retention. Run records are voluminous and their value decays unevenly — a week of detail supports incident work, a year of daily aggregates supports trend and capacity questions. Keeping everything forever is a cost; keeping too little can remove the baseline that data observability checks need when they compare against long-run trends or seasonality. Derive the retention period from the longest window any check, investigation, or audit has to look back over.
• Identity across systems. The same table named three ways by three tools produces three disconnected histories. Consistent naming is the unglamorous prerequisite for any of this being joinable.
• Where it surfaces. Operational facts are most useful next to the data they describe — a last-loaded time shown in the query editor beats the same fact in a monitoring tool nobody opens while writing a query.

One boundary is worth keeping clear: an audit log exists to evidence who did what for security and compliance purposes, and although query logs serve both, the retention rules and access controls an audit trail needs follow from that purpose and may differ from those you would choose for engineering telemetry. Where one log serves both purposes, the stricter requirement may have to apply to the whole log. Treat the overlap deliberately rather than letting one become the other by accident.

How the three kinds of metadata fit together, and why catalogs full of the technical kind go unused, is worked through in The Catalog Nobody Opens.

References

• OpenLineage Documentation
• dbt Documentation, Sources (source data freshness)