Column-Level Lineage
Column-level lineage records which specific fields a downstream dataset derives from, rather than only which tables it reads. The difference sounds like precision for its own sake and is the difference between lineage that answers a question and lineage that starts a conversation.
Consider a table of forty columns feeding a report that uses four of them. Table-level lineage says the report depends on the table, so every change to any of the forty columns appears to threaten it. Column-level lineage says the report reads order_id, order_date, amount, and status — and a change to the other thirty-six does not affect this report — provided the graph captured every way the report reads the table, including columns it filters or joins on without selecting, and provided the report is the only consumer. Unobserved is not the same as unused; what column lineage does is shrink the list of candidates to check, not certify the rest.
What it changes in practice
| Question | Table-level answer | Column-level answer |
|---|---|---|
| Can I drop this column? | Twelve datasets read this table — ask all of their owners | Two datasets read this column — ask those two |
| Where did this wrong figure come from? | One of these four upstream tables | This field is computed from these two upstream fields |
| Which reports contain personal data? | Anything reading a table that has a sensitive column | Candidates: anything that selects, computes from, or filters on the sensitive column — then a review of what each output reveals |
The third row is the one that turns a compliance exercise from an estimate into a list of candidates worth reviewing. Classification applied at table level over-reports by design, and over-reporting has a cost: controls placed on datasets that do not need them, and reviewers who learn to dismiss the output. Column lineage narrows the list; it does not decide it. A query that returns only person IDs but filters on a diagnosis column has produced a list whose membership reveals the diagnosis, though the sensitive column was never selected, and an aggregate that does select it may reveal nothing about anyone. Whether an output carries personal data is a judgment about identifiability and transformation; lineage supplies the dependencies that judgment needs.
Not all derivation is equal
A useful column-level graph distinguishes how a field was used, because the three cases carry different weight in impact analysis.
- Direct. The field is copied or renamed. A type change propagates immediately and visibly.
- Computed. The field is one input to an expression. A change may alter the result subtly — the worst case, because nothing fails.
- Filter or join condition. The field does not appear in the output at all but decides which rows do. Two changes here have different effects: dropping the source column usually makes the query fail with a missing-column error, while changing the condition or its inputs — a different threshold, a renamed status value, a join key that no longer matches — leaves the query valid and changes which rows are returned rather than a value, which is the quiet failure. Either way the dependency is invisible to anyone reading the output schema.
The third kind is the one people forget when reasoning by hand, which is a good argument for collecting lineage automatically rather than documenting it.
What it takes to get it
Table-level edges can be inferred from the fact that a job read one dataset and wrote another. Column-level edges require understanding what the job did. That understanding can come from parsing the transformation logic, from the engine’s execution plan, or from a mapping the transformation declares itself, and in each case the collector has to be able to see it. Lineage standards accommodate this by carrying such detail as extensions to the core model: in OpenLineage, “a facet is user-defined metadata and enables entity enrichment” on top of dataset, job, and run entities.
Where the logic is opaque, coverage degrades rather than failing loudly. The condition is what the collector can observe, which is not the same as how the code was written — a query assembled at run time still yields column edges if the collector sees the statement that was executed or the engine’s plan, and a transformation can also declare its own mapping. What actually defeats it is logic the collector never sees or cannot interpret: a user-defined function whose body is opaque to the parser, a step that moves a file wholesale, an operation the collector does not support. Those yield table-level edges at best. That makes partial coverage the normal condition, and the honest way to present it is to show which parts of the graph are column-level and which are not — a reader who assumes uniform precision will treat “no dependency found” as “no dependency exists,” which is the same error the general lineage coverage boundary warns about.
One boundary is worth stating because it changes what an impact analysis should return. A column can reach an output in two ways: by contributing its value, and by affecting which rows appear — a filter predicate, a join key, a window partition. Value dependencies are what most people picture, but a change to a filter column alters the result just as surely, so both belong in an impact analysis. What they do not deserve is the same label: distinguish them, because a reviewer reading a list of forty downstream columns needs to know which ones would change value and which would change population.
Two supplements make up most of the gap. Query logs show actual reads, including the ones no parser saw. And a data contract states which fields consumers depend on as a promise rather than an inference, which is the only version of this information that survives a consumer nobody has instrumented.
How lineage granularity fits with metadata types and the uses a catalog is actually for is worked through in The Catalog Nobody Opens.
References
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.