Data Platform Layers and Ownership

Open almost any data platform diagram and you will find boxes in a row: ingest, store, transform, serve. Below them, another row: bronze, silver, gold. The diagram is usually accurate as far as it goes. What it does not show is what any of it guarantees or who is answerable for it, which is what a reader needs. An analyst who asks “can I use this table?” is not asking which box it came out of. They are asking whether the numbers are right, whether they will still be right tomorrow, and whom to call when they are not.

This article is about turning those boxes into answers. It separates two things the diagram merges — the functions a platform performs and the states data passes through — explains what each data layer has to guarantee before the name means anything, works through where the popular layer naming misleads, and then takes on the question that decides how a platform scales organizationally: which data belongs to a central team and which belongs to the domain that produces it. The examples are invented for illustration.

Two different axes, drawn as one row of boxes

The confusion starts because the same diagram is asked to show two unrelated things.

Functional layers are the capabilities the platform provides: getting data in, storing it, transforming it, serving it, and managing it. They are about what the platform does, and they map to systems and to the teams that operate them.

Data layers are the states a given dataset passes through on its way from a source system to a consumer: raw as it arrived, cleaned and conformed, shaped for a particular use. They are about how far along a dataset is, and they map to tables and to the people accountable for their contents.

The two are orthogonal. Every data layer uses every functional capability: the raw layer is stored, transformed into the next layer, served to someone, and governed. Drawing them as one left-to-right flow implies that transformation happens once, between two boxes, which is exactly the misreading that produces platforms where nobody can say where a business rule lives.

Functional layer	What it provides	What breaks when it is weak
Ingestion	Getting data from sources reliably, with a record of what arrived and when. Covers batch extracts, change data capture, events, and files	Silent gaps and duplicates that surface as wrong numbers weeks later
Storage	Durable, queryable retention with a table abstraction, versioning, and retention policy	Cost surprises, unreadable history, and erasure requests that cannot be satisfied
Transformation	A place to express, test, schedule, and deploy the logic that turns one layer into the next, whichever of the ETL and ELT patterns it follows	The same business rule implemented three times with three answers
Serving	Delivering data in the shapes consumers need: SQL, extracts, APIs, feature values, files. Includes the arrangements that let one engine read another’s storage, such as external tables at the lake and warehouse boundary	Every consumer builds its own pipeline from the layer beneath
Management	Catalog, lineage, access control, quality checks, cost attribution, and the record of who owns what	Nobody can answer “where did this number come from” or “who may see this”

The management row is the one most often treated as a later phase. It is also the one that determines whether the other four can be operated by anyone other than their builders, which is why a platform without it tends to become the permanent responsibility of whoever set it up.

What a data layer has to promise

The data layers are worth defining by their guarantees rather than their contents, because the guarantee is what a consumer actually depends on.

Layer	What it promises	What it deliberately does not promise	Typical reader
Raw	What the source sent, in the representation and scope the platform chose to keep, unaltered apart from any stated preprocessing such as decompression, with a record of when it arrived and from where, for a stated retention period. That supports replay only as far as the code, schemas, reference data, and permissions a rerun needs are kept as well, and only once the replay path has actually been tested. Where it holds data only until the next step consumes it, it is a staging area rather than an archive: it makes neither the retention promise nor the replay promise, and the two have different retention needs	That it is correct, complete, deduplicated, or comparable across sources	Pipeline engineers, and anyone reconstructing history after a bug
Cleansed and conformed	Types are enforced, duplicates resolved, keys made consistent, and the same entity means the same thing across sources	That it answers any particular business question, or is shaped conveniently for one	Data engineers and analysts building on top
Serving	These are the agreed definitions of business measures, shaped for a use, with a stated freshness and an owner	That it contains everything, or that unusual questions can be answered from it	Analysts, dashboards, applications, models

Stated that way, the layers are not degrees of quality on one scale. They differ in which guarantee they offer, and the raw layer’s refusal to promise correctness is a feature: it is the platform’s own record of what the source said, which is often the most direct evidence when that is in dispute. It is not necessarily the only evidence, since the source may keep its own history or a copy of what it delivered.

A layer boundary becomes real when four things are true of it. Without these, the names are folder conventions.

• A stated contract covering two different things. Structure — field names, types, and whether a change is compatible with existing readers — can be checked automatically: a schema registry stores versioned schemas and validates evolution against a configured compatibility setting. A compatibility check cannot infer or guarantee meaning, because a registry sees no difference between an amount in dollars and the same field in cents. What a field denotes, the grain of a row, and what a null signifies have to be decided and written down by whoever owns the data. Once they are, much of it can be tested automatically — a fixture that checks a known unit conversion, a uniqueness test on the columns that define the grain, a rule for where nulls are allowed — and change review covers what tests cannot express.
• A stated freshness. Not “hourly” as an aspiration but a target with a measurement, since data freshness is what consumers silently assume when nobody states it.
• A named owner. Someone who decides what correct means for this dataset and is reachable when it is wrong, which is the distinction drawn in data owner, steward, and custodian.
• A change process. How a breaking change is announced, how long consumers have, and how the old shape is kept available in the meantime.

The practical test is to ask, for one table in each layer, who would be told if its meaning changed next week. Where the answer is “whoever notices,” that boundary is decoration.

Medallion, and what the name leaves out

The most widely used naming for these layers is the medallion pattern. Databricks describes it as a data design pattern for organizing data in a lakehouse, with the goal of incrementally improving the structure and quality of data as it moves through the layers.

• Bronze holds raw data from external sources, with table structures corresponding to the source systems as-is, plus metadata columns such as load date and time and the process that wrote it. Its stated focus is quick change data capture and keeping a historical archive of the source.
• Silver is matched, merged, conformed, and cleansed, producing what the description calls an enterprise view of key business entities and transactions. It emphasizes “just-enough” transformation and a model closer to third normal form, favoring speed and agility.
• Gold is consumption-ready and project-specific, de-normalized and read-optimized with fewer joins, carrying the final presentation-layer transformations and quality rules.

This is a sound pattern, and the same idea appears independently elsewhere. dbt’s recommended project structure uses staging, intermediate, and marts — where staging means persistent, source-conformed models that dbt suggests building as views, not the temporary load area the same word names elsewhere — and states the underlying principle in a way the metal names do not: the arc moves data from source-conformed to business-conformed, and stacking transformations in modular layers means each transformation is applied in only one place. That sentence is the actual point of layering. The medals are a label for it.

Two features of the naming cause trouble in practice. The first is that bronze, silver, and gold sound like grades of quality, which invites the reading that gold is the good data and bronze is the bad data. The layers differ in purpose, not merit: a raw record is exactly as it should be, and a gold table that answers the wrong question is not better for being gold. The second is that a three-medal ladder implies every dataset should climb it, which is not what the pattern requires and not what most platforms need.

The layer names leave several operational choices unresolved. The following failure modes show where the pattern needs a deliberate boundary, an owner, or a retention and replay decision.

Failure	What it looks like	What to do instead
Layers as a mandatory ladder	A reference table with twelve rows exists three times because “everything goes through all three”	Require layers where they earn their keep. A small, stable, single-source table can be one table
Silver as a dumping ground	Hundreds of tables, no consistent grain, nobody sure which are current	Treat conformed entities as a curated set with owners, not as everything that has been cleaned once
Business rules in the wrong layer	Revenue is defined slightly differently in four gold tables	Define each rule once, at the layer where it stops being source-specific, and reuse it
Gold per dashboard	Each request produces a new gold table, and the layer becomes a report archive	Distinguish shared business entities from one project’s presentation table, and let the second be cheap and disposable
Bronze as an unmanaged archive	It grows without retention, nobody reads it, and it cannot actually be replayed because the pipeline changed	Give it a retention policy and test the replay path, or accept that it is not a replay mechanism

The cost side is worth stating plainly, because layering is usually presented as free. What it costs depends on how the layer is built, and the two cases are different enough that collapsing them leads to the wrong conclusion.

• Materialized as its own tables, a layer is another copy to store, another job to run and monitor, and another hop of latency between the source and the consumer. dbt’s guidance on materializations puts the trade directly: a table is fast to query but has to be rebuilt and does not pick up new source data on its own.
• Expressed logically — as a view, or as dbt’s ephemeral materialization, which is not built in the database at all but interpolated into dependent models as a common table expression — the layer costs no storage and no refresh job of its own. A view is computed each time it is queried. dbt’s guidance describes a view as containing the latest records from its source data, which is the right way to read it: the layer adds no staleness, but it inherits whatever its inputs have. A view over a table refreshed nightly is exactly as current as that table. Ephemeral models need one more step in that reasoning, because they have no result of their own: their SQL is compiled into whichever model selects from them, so freshness follows how that model is built. Interpolated into a view, the logic runs at query time; interpolated into a table, the result changes only when that table is rebuilt. So the compute cost moves to wherever the logic runs: to every query for a view, and for ephemeral logic to the build of each downstream table or incremental model that includes it — paid again in every such model, since each compiles its own copy. The same guidance notes that views become slow when they carry significant transformation or are stacked on other views; ephemeral models also cannot be queried directly, which makes them harder to debug.

The distinction matters because the two questions are separable. Is this a real boundary — does it promise something the previous layer does not, with its own contract, owner, and response when it breaks? And should it be materialized — does it need a stored result? Query frequency and compute cost are part of that answer but not all of it. A stored result can also be what the layer promises: a validated version that stays fixed while its inputs change, reads that keep working while an upstream system is down, an access boundary, or a latency target. A layer that is rarely read and cheap to compute can still need storing for one of those reasons. A boundary worth having that needs none of them can become a view, once a view has been checked against the consumers’ query load, availability, and consistency needs. A layer that turns out to promise nothing the previous one does not is overhead in either form, and once it is clear that nothing depends on it, the honest move is to remove it rather than to make it cheaper. Having several consumers makes a boundary easier to justify, but it is not the test: a single consumer that depends on a stated guarantee, or a step that exists to isolate one team from another’s changes, is a real boundary with one reader.

Where transformation actually belongs

The recurring argument on data teams is not whether to have layers but which rule goes in which one. A workable test is to ask what the rule depends on.

• Depends on the source system’s quirks — a column that encodes status as single letters, a timezone the source records in local time, a legacy identifier format. This belongs at the first transformation, because the knowledge is about the source and nothing downstream should need it.
• Depends on identifying an entity across systems — what counts as one customer when three systems each have a record, or which order records refer to the same order. This belongs in the conformed layer, because every domain that joins to it needs the same answer. Note that this is identity, not measurement: when an order counts as recognized revenue is an accounting definition, and it belongs wherever that definition is owned, which is usually finance rather than the shared layer.
• Depends on one use — a dashboard’s chosen comparison period, a model’s feature window, a report’s rounding convention. This belongs in serving, and should be easy to change without disturbing anyone else.

The second category needs one qualification, because it is easy to over-apply. Identity has to be connected: if two domains cannot tell which of their records refer to the same customer, nothing joins reliably. That does not require one key or one entity model everywhere. A shared identifier is one way; a governed mapping between, say, a CRM customer ID and a billing account ID is another, provided it states its cardinality, the period each link is valid, and how unmatched or uncertain matches are handled. Some domains model different units — a person, a household, a legal entity — and need a stated relationship between them rather than a single definition. Measures computed about those entities often should not be shared, because different domains legitimately measure different things. What has to be common is a managed way to connect the entities and a stated meaning for each; what each domain does with them can differ as long as it says so.

The rule that is usually placed wrong is the second. Entity definitions pushed into serving get re-implemented per consumer, and the failure that follows is a specific one: four teams set out to implement the same definition, write four slightly different versions of it, and the same question returns four answers. That is not the same as four domains reporting different revenue figures because they deliberately measure different things, which is legitimate as long as each states what it measures. The first is drift in one definition; the second is several definitions, and only the first is a defect. Pushed too early, into the raw layer, they destroy the ability to recover the source’s actual statement, which is the one thing the raw layer exists to preserve. Neither mistake is visible on a diagram. A reconciliation against the source catches part of it — keys, values, counts, and totals that disagree — but not all. A copy whose counts and totals match the source can still count one person’s two local IDs as two customers; the source may hold no correct entity to compare against; and a raw layer that was transformed too early may no longer hold the evidence. Entity errors also need fixtures built from the approved definition, with expected results worked out independently of the implementation.

Shared data and domain data

Layers organize data. They do not say who is responsible for it, and that is the question that decides whether a platform keeps working as an organization grows.

The default arrangement has one central data team own every transformation. It works at small scale and can become a bottleneck as the organization grows, for reasons that are structural rather than a matter of effort — where they hold. If the central team is the only group permitted to change the data, lacks the domain knowledge to judge a change, and works through one queue, every domain’s change waits behind every other domain’s. A central team with domain specialists, parallel work, and delegated change rights avoids much of that. Choosing between central, federated, and domain ownership is a comparison of capabilities and of the paths changes actually take, not of organizational labels; a growing queue of domain requests and changes blocked on missing context are the signs to look for.

Zhamak Dehghani’s data mesh writing is the best-known articulation of the alternative, built on four principles: domain-oriented decentralized ownership, data as a product, self-serve data infrastructure as a platform, and federated computational governance. Two of them bear directly on layer boundaries.

Data as a product means a dataset published for others carries obligations, not just bytes. Her list of required characteristics includes discoverability, security, explorability, understandability, and trustworthiness, with a data product owner accountable for them. That is the same set of conditions as the layer contract above, applied to a team’s output, and it is the reason data product is a useful term rather than a rebranding of “table.”

Federated computational governance answers the question that decentralization otherwise leaves open. Some decisions have to be global or nothing interoperates — her example is how to identify a podcast listener consistently — while modeling within a domain stays local. The word computational is doing work: global policies are meant to be embedded in the platform and enforced automatically rather than published as guidance that teams are asked to follow.

Adopting the full model is a large organizational change and not the only option. The useful part for any platform, mesh or not, is the question of which decisions are global. Three tests separate them.

• Does more than one domain have to agree for the data to join? How customer and product records are identified or mapped across domains, currency and time handling, and the identity of shared entities are global, because unmanaged divergence there makes the data unusable together. Global means a shared key or a governed mapping, not a single model, and it is narrower than it sounds: agreeing on who a customer is does not require agreeing on what makes one active.
• Does an outside obligation attach to it? Classification of personal data, retention periods, and access rules follow from regulation and contracts, and cannot be set per team.
• Would divergence be invisible until it caused harm? The problem is not that two measures differ but that they carry the same name and nobody knew. Marketing counting anyone who opened the app and finance counting anyone who paid are both legitimate definitions of an active customer, and forcing them to agree would make one of them useless. Two numbers differing is not by itself a defect. It becomes one when both are published as “active customers” with no qualification, so that a slide combines them or a reader assumes they are comparable. The remedy is naming, scope, and a stated definition — not a single global number.

Everything failing all three tests should stay with the domain. Pulling those decisions to the center risks the bottleneck decentralization was meant to remove, and adds the cost of a group deciding things it may not have the context to decide well.

Who owns which layer

Combining the two axes gives a division that survives contact with an incident. The point of writing it down is that during an incident nobody negotiates; they follow whatever was already agreed.

Layer	Platform team owns	Domain or producing team owns	Consumer owns
Raw	The ingestion mechanism and retention; that what it received was stored durably; and, within the agreed ingestion window, reconciling what arrived against what the producer says it sent and re-ingesting gaps while the source can still supply them	The source contract: what is emitted, its schema, notice before it changes, sending reliably with retries and keeping records until delivery is confirmed, and enough information to tell whether a record was emitted at all	Nothing; consumers should not read it for production use
Cleansed and conformed	The transformation runtime, scheduling, testing framework, and lineage	The rules themselves, what correct means for its entities, and the response when a check fails	Raising a defect against a stated contract
Serving	Access mechanisms, performance, cost attribution	Shared business definitions and their freshness targets	Use-specific shaping, and its own dashboards and features
Management	Catalog, policy enforcement, quality tooling, and the audit trail	Populating it: ownership, classification, and descriptions for its own data	Discovering through it rather than by asking colleagues

The raw and management rows show why delivery and ownership need explicit agreements. In the raw row, the platform’s guarantee is about delivery, not content. If the source emitted a wrong value, the platform stored a wrong value correctly, and the defect belongs to the producer. Making this explicit prevents the common failure in which the platform team becomes the debugger of everyone else’s source systems.

Delivery, though, is more than storing what showed up, and “we stored what arrived” says nothing about a record the source emitted and the platform never received. That record can be lost at several points — the producer’s send, the broker or connector, the platform’s own write — and responsibility changes hands at each. A producer that sends without waiting for acknowledgement, or discards its buffer before delivery is confirmed, keeps the obligation to retry and retain; Kafka, for instance, documents that with acks=0 a record counts as sent with no guarantee the server received it. The platform owns its connectors, offsets, and schedules, so it is well placed to watch its side of the handoff. That is worth stating as a specific obligation: for an agreed window, detect missing or late data by tracking completeness against something the source can confirm — a sequence, a log position, a manifest or outbox on the producer’s side, or a count for the period — and reconcile with the producer. Re-ingestion is possible only while the source or broker still holds the history. For gaps found after that, the agreement should say how recovery works and how consumers are told about a loss. None of this guarantees that the data is complete at the source, which the platform cannot know and should not claim.

In the management row, the split between providing a catalog and populating it is where most catalog programs fail. A platform team can deliver a complete tool and still end up with a catalog nobody trusts, because ownership and descriptions were left to be filled in by people with no reason to do it. If domains are expected to populate it, the expectation has to be stated, made easy, and checked — catalog coverage exists as a metric because the alternative is assuming.

Questions that reveal a weak layer boundary

Question	If the answer is unclear
What does this layer promise that the one before it does not?	The layer may be a copy rather than a boundary. Before removing it, find what depends on it and whether it quietly provides something — reads that keep working when an upstream system is down, a validated fixed version, an access boundary, a latency target. Remove it only once nothing needs those
Who is told when a table here changes meaning, and how far in advance?	Consumers find out from a broken dashboard
Who decides what correct means for this data?	Defects route to whoever runs the pipeline rather than whoever knows the answer
Which business rules live here, and where else are they implemented?	The same measure is computed differently in several places
What is the stated freshness, and is it measured?	Consumers assume a number nobody promised
Which definitions here must be identical across domains?	Teams diverge quietly and reconcile in a meeting
Is this layer materialized, and if so why is a view not enough?	Storage and refresh cost may be paid for a boundary that could be logical. Replace it with a view only after checking that a view meets the consumers’ query load, availability, and consistency needs

References

The layer patterns cited are design conventions rather than product features and are described by their authors. The Databricks glossary entry defines the medallion pattern; use the failure-mode comparison above when deciding where its layers need additional operating choices.

• Databricks Glossary, Medallion Architecture
• dbt Developer Hub, How We Structure Our dbt Projects
• Zhamak Dehghani, Data Mesh Principles and Logical Architecture (martinfowler.com, 2020)
• dbt Developer Hub, Materializations
• Confluent Documentation, Schema Registry
• dbt Developer Hub, Staging: Preparing Our Atomic Building Blocks
• Apache Kafka Documentation, Producer Configs: acks