Metadata and Lineage for Change Impact Analysis

An illustrative case: a platform team wants to add a new payment status, partially_refunded, to a column in a source table called payments. The change is additive on paper, since it adds a value rather than a column. Nobody doubts that it is a reasonable request. The team still has to answer a question that sounds simple and is hard in practice: which datasets, reports, features, and exports will behave differently once the new value starts to appear?

Answering it from memory fails in a predictable way. The people who built the first dashboards have moved on, a data scientist copied the status column into a feature table last year, and a finance spreadsheet runs a query once a month. The only reliable alternative is a recorded description of how data flows between assets, together with an honest estimate of how complete that description is. That description is the subject of this post: metadata that is captured as a side effect of running pipelines, the lineage graphs built from it at table and column granularity, the measurement of coverage, the discovery of consumers the graph does not know about, and the validation of a change before it ships.

The previous post, Data Quality Invariants and Business Reconciliation, covered how to decide whether a dataset conforms to its rules and agrees with a reference. This post takes the complementary question. When a rule, a definition, or a column changes, those checks tell you whether the new data conforms, and lineage tells you whom the change reaches. The post assumes you know what a pipeline stage and a table are and need not have used a lineage tool.

What metadata is and where it comes from

Metadata is data about data. Three families are useful to keep apart because they come from different places and answer different questions.

Family	Describes	Example	Typical source
Technical	Shape and location	Column names, types, partitions, storage location	Inspecting a system at rest, such as an information schema
Business	Meaning and responsibility	Definitions, owners, classifications, permitted use	People declaring it, then tooling storing it
Operational	What happened	Last load time, row counts, run status, who queried what	Emitted by runs or read from logs

The data catalog entry describes how a catalog organizes the first two families so that people can discover resources, and the operational metadata entry describes the third family. Lineage draws on all three: it is a relationship among technical objects, observed through operational events, and interpreted with business metadata such as an owner or a classification.

Five ways to capture it

A lineage or metadata platform gets its information by some combination of five methods. They differ in what they can see and in what they miss, and a platform’s coverage is the union of the methods it actually runs.

Method	How it works	Sees	Misses
Scan	Periodically inspect catalogs, information schemas, and storage	Names, schemas, current location	Anything that is not registered; history between scans; usage
Emit	The job or orchestrator reports reads and writes as it runs	Real inputs and outputs of real runs	Jobs without an integration; paths not executed in that run
Parse	Analyze SQL or code to extract table and column dependencies	Intent in the statement, including branches not executed	Dynamic SQL, opaque functions, logic outside the parser’s dialect
Declare	People or configuration state the dependency or owner	Relationships no tool can infer, such as an external consumer	Anything nobody wrote down; declarations that went stale
Observe	Read query logs and access history	Actual reads, including by consumers nobody registered	Reads outside the retention window; reads through shared identities

No single row is enough. Emission sees real runs but only for integrated jobs. Parsing sees code but not what ran. Observation sees consumers but only within a retention window. The practical consequence is that a lineage graph is a union of evidence with differing reliability, and each edge should be able to say how it was learned.

An open event model, read from its documentation

OpenLineage is an open specification for lineage metadata. Its documentation, read on 3 October 2026 and showing version 1.53.0, describes an object model with three core entities and a set of event types. A job is a process that consumes or produces datasets, and the documentation says it is abstract and may map to a workflow task, a model, a query, or a checkpoint. A run is one occurrence of a job in time, identified by a run identifier that the client generates and keeps constant across the events of that run. A dataset is a discrete collection of data, such as a table, with a name that is unique within a namespace derived from its physical location. Each of the three can be extended by facets, which are additional named metadata.

The documentation defines three kinds of event. A run event describes the execution of a job and is emitted at runtime. A job event describes static properties such as source location or declared inputs and outputs, and a dataset event describes metadata changes such as schema or ownership; both of those are design-time events, not tied to a run. The run cycle defines six states: START, RUNNING, COMPLETE, ABORT, FAIL, and OTHER, where COMPLETE, ABORT, and FAIL are terminal. The run-cycle page also says that batch jobs are expected to finish, while streaming jobs have no natural completion and typically report periodically.

Two details matter for impact analysis. First, because a dataset is identified by namespace and name, the same physical table named differently by two tools becomes two nodes with two histories, so consistent naming is a prerequisite for a connected graph. Second, a run event carries both the inputs and outputs of a specific run, so the graph can be woven from observations of many jobs, and it can answer which run produced a particular output. A system that stores only the current state cannot say what was true on a past date.

The following JSON sketch shows the shape of a run event for the payments job. It is a simplified, illustrative sketch that omits required fields and is not validated against the specification. Consult the specification for the exact schema.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-03-02T01:14:07Z",
  "run":  { "runId": "<uuid-generated-by-client>" },
  "job":  { "namespace": "warehouse", "name": "build_fct_payments" },
  "inputs":  [ { "namespace": "warehouse", "name": "stg.payments" } ],
  "outputs": [ { "namespace": "warehouse", "name": "fct.payments" } ]
}

The event says that one run of a job read one table and wrote another. That is table-level lineage, learned by emission. Nothing in it says which columns mattered, and for that the model needs a facet.

Pipeline lineage and column lineage

Table-level and pipeline-level edges

The coarsest useful graph has datasets and jobs as nodes, and edges that say a job read a dataset or wrote one. This graph is easy to obtain, because it needs only the observation that a run touched a dataset. It answers questions such as which jobs run downstream of a table. Its weakness is over-reporting. The data lineage entry describes the basic use, and the column-level lineage entry describes why granularity matters: a report that reads four of forty columns appears to depend on all forty at table level, so every change to any column looks threatening, and reviewers learn to ignore the output.

Column-level edges and their types

Column-level lineage records which input columns contribute to which output columns. The OpenLineage column lineage facet, as documented, lists for each output field a set of input fields, and each input field carries a transformation with a type, a subtype, a description, and a masking flag. The types are worth knowing because they map directly onto impact reasoning.

Type	Subtype	Meaning in the documentation	What a change to the input does
DIRECT	IDENTITY	Output value taken as is	A type or value change appears in the output
DIRECT	TRANSFORMATION	Output is a transformed source value	The result may change in non-obvious ways
DIRECT	AGGREGATION	Output aggregates values from many input rows	The aggregate changes; one bad row is diluted but not removed
INDIRECT	JOIN	Input is used in a join condition	Which rows are matched changes
INDIRECT	FILTER	Input is used as a filter, such as a WHERE clause	Which rows appear changes
INDIRECT	GROUP_BY, SORT, WINDOW	Input determines grouping, ordering, or windowing	Grouping, order, or partitioning changes
INDIRECT	CONDITIONAL	Input used in IF, CASE WHEN, or COALESCE	Which branch executes changes

The documentation’s own distinction is that for DIRECT the output value is derived from the input, and for INDIRECT the output is influenced by the input without being derived from it. A column that is used only in a filter does not appear in the output schema at all, so a reader of the output cannot see the dependency, yet changing it changes the result as surely as changing a selected column does. That is why an impact analysis should return both kinds of edge and label them differently: a reviewer reading forty downstream columns needs to know which would change in value and which would change in population.

A worked example on an illustrative graph

The following edge list is an illustrative graph for the payments scenario, written in the shape of OpenLineage’s types and subtypes. It is not the output of a real system. The traversal below starts from the source column that will receive the new value, follows edges downstream, and labels each reached column value when it was reached only through DIRECT edges from a value-carrying column, and population otherwise. The code is a simplification: a node first reached as population is not re-expanded if a later path would make it value. That is acceptable for this small graph and would need a proper fixpoint in a real implementation.

from collections import deque, defaultdict

# (source column, target column, type, subtype)
EDGES = [
    ("src.payments.status",      "stg.payments.status",       "DIRECT",   "IDENTITY"),
    ("src.payments.amount",      "stg.payments.amount_minor", "DIRECT",   "TRANSFORMATION"),
    ("src.payments.card_brand",  "stg.payments.card_brand",   "DIRECT",   "IDENTITY"),
    ("stg.payments.status",      "fct.payments.is_captured",  "DIRECT",   "TRANSFORMATION"),
    ("stg.payments.status",      "fct.payments.amount_minor", "INDIRECT", "FILTER"),
    ("stg.payments.amount_minor","fct.payments.amount_minor", "DIRECT",   "IDENTITY"),
    ("stg.payments.card_brand",  "fct.payments.card_brand",   "DIRECT",   "IDENTITY"),
    ("fct.payments.amount_minor","mart.revenue_daily.revenue","DIRECT",   "AGGREGATION"),
    ("fct.payments.is_captured", "mart.revenue_daily.revenue","INDIRECT", "FILTER"),
    ("fct.payments.card_brand",  "mart.card_mix.share",       "DIRECT",   "AGGREGATION"),
    ("fct.payments.is_captured", "feat.customer_risk.capture_rate","DIRECT","AGGREGATION"),
    ("mart.revenue_daily.revenue","bi.exec_dashboard.revenue_tile","DIRECT","IDENTITY"),
]
out = defaultdict(list)
for s, t, ty, st in EDGES:
    out[s].append((t, ty, st))

def impact(start):
    seen = {start: "origin"}
    q = deque([start])
    while q:
        c = q.popleft()
        for t, ty, st in out:
            kind = "value" if (ty == "DIRECT" and seen in ("origin", "value")) else "population"
            if t not in seen:
                seen[t] = kind
                q.append(t)
            elif seen[t] == "population" and kind == "value":
                seen[t] = "value"
    return seen

res = impact("src.payments.status")
for col, kind in sorted(res.items()):
    print(f"{kind:10} {col}")
tables = {c.rsplit(".", 1)[0] for c in res} - {"src.payments"}
print("column-level assets:", len(tables))
all_down = {t.rsplit(".", 1)[0] for _, t, _, _ in EDGES}
print("table-level assets downstream of src.payments:", len(all_down))

Running this snippet printed:

population bi.exec_dashboard.revenue_tile
population fct.payments.amount_minor
value      fct.payments.is_captured
value      feat.customer_risk.capture_rate
population mart.revenue_daily.revenue
origin     src.payments.status
value      stg.payments.status
column-level assets: 5
table-level assets downstream of src.payments: 6

Read the output in three steps. First, the table-level count is 6 and the column-level count for the changed column is 5. The sixth asset, mart.card_mix, reads only card_brand, which the status change never touches, so column-level lineage removes it from the candidate list. With 6 and 5 the saving is small, but in a table with forty columns feeding many reports the same mechanism is the difference between asking twelve owners and asking two.

Second, the new status value changes the value of is_captured and of feat.customer_risk.capture_rate. A transformation that maps status strings to a captured flag has to decide what partially_refunded means, and a feature computed from that flag will shift. These are the silent cases, because nothing fails.

Third, revenue and the dashboard tile are labeled population. The revenue measure is computed from amount_minor, which is not derived from status, but the status decides which rows are included. Revenue can change because the set of rows changes, and the change appears in the dashboard even though no revenue formula was edited.

The graph does not say what the new value should do in each place. It supplies the list of places where someone must decide, and the decisions are a human job with an owner per asset.

Limits of the graph

Several things limit what such a graph can tell you.

• Parsing has a reach. Edges come from what the collector can observe or interpret. The column-level lineage entry notes that where the logic is opaque, such as a user-defined function the parser cannot read, or a step that moves a file wholesale, coverage degrades to table-level edges at best, without failing loudly.
• Executed paths and declared paths differ. An emitted edge reflects a run, and a branch that did not execute in that run leaves no edge. A parsed edge reflects the code, including branches that never run in practice. Prefer a graph that stores which method produced each edge and when it was last confirmed.
• Semantic changes are invisible to structure. If a column that held dollars starts to hold cents, every edge is unchanged. Lineage can tell you who reads the column but not that the meaning changed. Contracts and definitions carry that meaning, and the schema evolution and compatibility entry lists examples where structural validation passes while metrics become wrong.
• Lineage describes dependency, not correctness. A perfectly recorded pipeline can faithfully propagate a wrong source value.

Coverage: how much of the graph do you actually have

Impact analysis fails in the direction that matters most when the graph is incomplete, because a missing edge reads as no dependency. Coverage measurement turns that risk into a number you can reason about. As with catalog coverage, the denominator must come from an inventory outside the lineage store. If the denominator is the set of assets that the lineage tool already knows, an unregistered table is missing from both counts and the less the tool knows the better the figure looks.

Three coverage measures are worth separating.

• Node coverage. The share of in-scope assets, such as production tables, dashboards, and models, that appear in the graph at all.
• Edge coverage. The share of real dependencies that the graph records. This is the one that matters for impact analysis and the hardest to measure, since the true edge set is unknown. Observed reads in query logs are one imperfect reference.
• Granularity coverage. The share of edges that are column-level rather than table-level, and the share that carry a type and subtype.

A calculation with assumed numbers

Suppose a warehouse has 500 production tables. The lineage store contains 430 of them, so node coverage is 430 / 500 = 86 percent. Of the 430, column-level edges exist for 260, which is 260 / 500 = 52 percent of all tables, not 60 percent of the 430. Now compare the graph with the query log for the last 90 days. The log shows 1,200 distinct pairs of (reading consumer, table). The graph explains 960 of them as an edge or a path. Treat the log as a reference, and edge recall against observed reads is 960 / 1,200 = 80 percent. The remaining 240 reads are consumers or paths the graph did not know: a notebook, a spreadsheet connection, a service that queries with a shared account. These numbers are assumptions chosen to make the arithmetic clear, not measurements.

Two cautions apply to that 80 percent. First, it measures recall only against reads the log recorded, so it cannot see consumers outside the log’s window or access path. A consumer that runs once a quarter is invisible to a 30-day window and visible to a 90-day window only if its run falls inside it; to see a consumer that runs on a period of \(T\), the window has to be at least \(T\). Second, a precise-looking percentage can hide a skewed distribution. Weight coverage by importance, such as by read volume or by criticality of the consumer, because 80 percent of edges can coincide with 40 percent of the edges that matter.

Reconcile the graph against observation

This is the same technique as reconciliation in the previous post, applied to metadata. Treat the lineage graph as one side and the access history as the other, and break down the difference: reads with no graph edge are coverage gaps; edges with no reads in the window are either unused or infrequent; edges whose reads come only from a shared service account need resolving to a person or report by query tags or BI metadata. Keep the result as a list with owners rather than a single percentage. The operational metadata entry notes that a shared service account must still be resolved to a report or a person.

Consumer discovery

A change affects consumers, not tables. Consumer discovery is the work of building the list of who reads an asset, and it combines sources that see different things.

Source	What it finds	Blind spot
Lineage graph (emit and parse)	Pipeline and model consumers	Anything outside integrated tools
Query and access logs	Real readers, including ad hoc ones	Reads outside the retention window; shared identities
BI and application metadata	Dashboards, reports, extracts	Local copies and downloads
Declared consumers and contracts	Consumers that registered a dependency	Anyone who did not register
Code and repository search	Hard-coded references in jobs and notebooks	Code in tools you cannot search
Direct questions to owners	Knowledge of external or offline use	Departed owners; forgotten uses

The data contract entry makes a point that explains why declared consumers matter even when logs are available: a stated promise survives a consumer nobody has instrumented, while an inference does not. A practical arrangement is to treat the logs as the primary discovery source, the graph as a structured view of what it can parse and emit, and registration plus a notice period as the mechanism that gives a consumer a reason to appear.

A short query illustrates the log-based approach. This SQL is illustrative and uses generic table and column names, not the schema of a specific product; many warehouses expose some form of query or access history, with different names, contents, and retention.

-- Illustrative: distinct readers of one table over a window.
SELECT user_or_service_account,
       query_tag,
       count(*)            AS reads,
       max(query_start)    AS last_read
FROM   query_history
WHERE  query_start >= current_date - INTERVAL '90' DAY
  AND  referenced_tables LIKE '%fct.payments%'
GROUP  BY 1, 2
ORDER  BY reads DESC;

Read the result as a candidate list. A row with a service account and no tag has to be resolved to an owner. A row with few reads and a recent last_read may be a monthly job. The absence of a row proves only that no read occurred in the window through an access path the log records.

Validating a change

Change validation connects the graph and the consumer list to an actual decision about whether and how to proceed. Different changes need different checks, and a table of change types is a useful way to see what the graph can and cannot tell you.

Change	What breaks	What the graph shows	What else is needed
Add a nullable column	Positional readers, strict schemas	Readers of the table	Test actual readers and exports
Rename or drop a column	Queries that name it	Every consumer of that column	Parse and compile downstream; stage the change
Widen a type	Casts and strict readers	Consumers of the column	Run downstream tests
Add an enumerated value	Code that handles known values	Direct and filter consumers	Review each branch and filter
Change a unit or definition	Values, silently	The same edges as before	Contract, definition change record, restatement decision
Change a filter or join	Population, silently	Indirect edges	Compare outputs before and after
Backfill history	Previously published numbers	Downstream aggregates	Announce as restatement

The last three rows share a property: the graph identifies where to look, but it cannot judge whether the change is acceptable. The procedure below separates the two jobs.

1. Compute the affected set from the changed column, following both DIRECT and INDIRECT edges, and keep the edge type on each result.
2. Add consumers the graph does not know, from the discovery sources above. Record which assets have no column-level edges and treat them as unknown rather than unaffected.
3. Validate mechanically. Rebuild or compile every affected asset against the changed input and run its tests. dbt offers a concrete example of selecting affected work: with the --state option pointing at a previous manifest, state:modified selects new and changed nodes, and placing the + graph operator after a selection includes the selected nodes and all their descendants, so a selection such as state:modified+ runs the changed models and everything downstream of them (documentation read on 3 October 2026). This validates that code still compiles and tests pass. It does not validate meaning.
4. Compare outputs. For each affected asset, compute the output with the old and new input over a fixed set of data and report differences by type: unchanged, changed values, added or removed rows. The comparison is the evidence that separates a harmless addition from a population change.
5. Notify and decide per owner. The list of owners and the measured differences are the content of the notice, in the same way that a metric change should be announced with its measured delta. The metric change management entry describes that practice.
6. Stage the change. For incompatible changes, the expand and contract pattern gives each consumer time to move, and the old form is removed only after use of it has stopped.

A comparison query for step 4 can be written with set operations. This SQL is illustrative and assumes both versions of the output exist as tables with the same key.

-- Rows only in the new output, only in the old output, and rows with changed values.
SELECT 'added'   AS change, n.* FROM new_out n LEFT JOIN old_out o USING (day) WHERE o.day IS NULL
UNION ALL
SELECT 'removed' AS change, o.* FROM old_out o LEFT JOIN new_out n USING (day) WHERE n.day IS NULL
UNION ALL
SELECT 'changed' AS change, n.* FROM new_out n JOIN old_out o USING (day)
WHERE n.revenue IS DISTINCT FROM o.revenue;

The IS DISTINCT FROM comparison treats two NULLs as equal and a NULL and a value as different, which matters because an ordinary inequality returns unknown when either side is NULL and would hide a change.

Validating the graph itself

A graph that is trusted for impact analysis should be tested like any other control. Two techniques are useful. The first is a seeded change: add a harmless column or filter to a test copy of a pipeline, run the lineage collection, and check that the graph shows the new edge with the right type. If it does not, you have found a blind spot in the collector. The second is the reconciliation described above, comparing edges with observed reads. Both are samples; a passing result supports the claim that the collector handles those patterns, not that it handles all patterns.

Operating the metadata

Capturing lineage is easier than keeping it useful. Four operating questions recur.

• Freshness and history. Lineage that reflects only the current state cannot reconstruct which run produced a number a report used. Keep a record per run for a retention period that matches how far back investigations need to go. The operational metadata entry discusses the retention trade: a week of detail supports incident work, and longer summaries support trend questions.
• Ownership. An asset with no accountable owner cannot receive a change notice. The data product owner entry describes an owner who can negotiate scope and decide on deprecation. Without a person who accepts the notice, a notice list is a list of addresses.
• Noise and ranking. In a connected graph, a change to a widely used dimension appears to affect most of the warehouse. Rank candidates by edge type, distance from the change, read volume, and criticality, and show the reason for each rank, so that a long list is usable. Ranking is a heuristic that reorders the list; it does not remove items from the obligation to check.
• Access to metadata. Metadata can itself be sensitive, because a lineage graph reveals where personal data flows, and the catalog entry for a dataset is not a grant of access to its rows. Control who can see lineage and who can see data separately.

Failure modes

The most important failure is a false negative caused by an incomplete graph, followed by an over-confident statement such as “nothing downstream uses this.” Phrase results as “no consumers found among these sources over this window.” A second failure is a stale graph, where a renamed table keeps its old node and edges so impact analysis returns results for an asset that no longer exists. A third is schema-only lineage that ignores filters and joins, which undercounts the impact of changes to filter columns. A fourth is treating the graph as authoritative when it was built by parsing code that has since been replaced. A fifth is the opposite problem: so many table-level edges that every change looks risky and reviewers stop reading the output.

Where this leaves you

Metadata capture gives you evidence of differing reliability. Lineage graphs organize that evidence, and column-level edges with types let an impact analysis separate value changes from population changes and drop assets that read other columns. Coverage measurement, reconciled against observed reads, tells you how far to trust a statement of the form “no downstream consumers.” Consumer discovery adds the readers the graph cannot see, and change validation combines the affected set with mechanical checks and output comparisons. The questions the graph cannot answer, such as what a new value should mean in each consumer, remain decisions with named owners.

The next post, Master Data and Entity Resolution Pipelines, applies these ideas to a harder case: a derived record assembled from several sources by rules, where every merge has to be traceable and reversible, and where the lineage from master record back to source records is what makes a correction possible.

References

• OpenLineage: Object Model (read 3 October 2026; version 1.53.0 shown)
• OpenLineage: The Run Cycle (read 3 October 2026)
• OpenLineage: Column Level Lineage Dataset Facet (read 3 October 2026)
• dbt Documentation: Node selector methods (read 3 October 2026)
• dbt Documentation: Graph operators (read 3 October 2026)