Storage, Compute, and Metadata: What Each Layer Actually Owns

A team moves its analytics off a warehouse product and onto object storage. The migration plan says “store the data in Parquet and query it with an engine,” and that sentence turns out to hide four more decisions. Which table format defines what a table is. Which catalog tells an engine where to find it. Which engine, and whether a second one can read the same data. And what happens when two jobs write at once — something the warehouse handled without anyone asking.

The warehouse was one product doing five jobs. Taking it apart is what makes open formats and independently scaled compute possible, and it is also what makes these five decisions yours. This article works through what each component owns, what object storage genuinely cannot do on its own, how a table format closes that gap, what schema evolution and deletes actually cost underneath, and where the copies come from once data can move freely. The examples are invented for illustration.

Five components, five different jobs

The pieces are easy to confuse because they are usually named together. Separating them by what each one is responsible for makes the dependencies visible.

Component	What it owns	What it knows nothing about
Object store	Durably holding bytes under a key, and access control on those keys	Rows, columns, tables, or which objects belong together
File format	The layout inside one file: columns, row groups and column chunks, encodings, per-chunk statistics	Any file other than itself
Table format	Which files constitute a table right now, its schema history, and how a change becomes visible atomically	How to execute a query
Catalog	Where the current definition of a named table lives, and often who may read it	The contents of the files
Engine	Planning and executing queries against whatever the other four expose	Nothing durable; it can be replaced without moving data

The column on the right is the useful one. A file format cannot answer “is this file still part of the table,” which is why a directory of Parquet files is not a table. A table format cannot answer “where is the table called orders,” which is why a catalog exists. And the engine, which is the part teams argue about most, is the one component that owns nothing permanent — which is precisely the benefit the separation was after.

What an object store will and will not do

Object storage is often described as “like a file system but cheaper,” and the differences that matter are the ones that description hides. Amazon S3’s documented consistency model, checked in September 2026, is a good specific case.

What it guarantees is stronger than many people assume. It provides strong read-after-write consistency for PUT and DELETE of objects in all regions, covering new objects, overwrites, and deletes. A read issued after a successful PUT returns what the PUT wrote, and a listing issued after that PUT includes the new object. Updates to a single key are atomic: a concurrent reader gets either the old data or the new, never a partial object.

Two documented limits are what the rest of the stack exists to work around. The first: S3 does not support object locking for concurrent writers, and if two PUT requests are made simultaneously to the same key, the request with the latest timestamp wins. The second, and the more consequential for tables: updates are key-based, and there is no way to make atomic updates across keys.

The first limit has a documented escape that is worth knowing precisely, because the next section depends on it. A writer can attach a precondition to the write: If-None-Match: * succeeds only if no object with that key exists, and If-Match succeeds only if the object’s ETag still matches the one the writer saw. A failed precondition returns 412 Precondition Failed, and when several conditional writes race for the same key, the documentation states that the first to finish succeeds and S3 fails the rest. So a single key can be claimed by exactly one writer, which is a compare-and-set on one object. What remains absent is the second limit: no precondition spans two keys.

A table spanning forty objects therefore has no way, at the storage layer, to change all forty at once. A job that rewrites twenty and fails leaves a directory that is neither the old table nor the new one, and a reader listing during the write sees a mixture. Note that this is not a consistency weakness of the store — each individual object behaved correctly — but an absence of the operation a table needs, which is a transaction over a set of objects.

Two more properties shape everything above. In general purpose buckets there is no rename: an object’s key is its identity, so “moving” data means copying and deleting, at full cost. The exception is narrow but real — directory buckets using the S3 Express One Zone storage class support a RenameObject operation that renames atomically without data movement, typically in milliseconds regardless of object size, and AWS documents that storage class as the only one supporting it. Treat rename as available where you have deliberately chosen that bucket type, and absent everywhere else. And listing is an operation with a price, proportional to the number of keys, which is why a table with a hundred thousand small objects is slow to plan against before it is slow to read.

How a table format publishes a version

The table format’s entire job is to supply the missing operation: make a set of files become the table, all at once. Both major formats do this by reducing a multi-file change to one atomic publication point — a single place whose change is what makes the new version current. They differ both in what that point is and in what performs the atomic act on it.

Iceberg keeps a hierarchy. A table metadata file records the current snapshot, the schema history, partition specs, and properties. Each snapshot has an ID and a monotonically increasing sequence number, and points to one manifest list. The manifest list names the manifests in that snapshot, with partition bounds that let whole manifests be skipped. Each manifest is an Avro file listing either data files or delete files — never both — with each entry carrying partition values, column statistics, and a status of added, existing, or deleted. At the bottom are the data files themselves.

The commit is the top of that hierarchy. The specification states it directly: an atomic swap of one table metadata file for another provides the basis for serializable isolation, and readers use the snapshot that was current when they loaded the table metadata, unaffected by changes until they refresh. A writer builds new metadata assuming the version it started from is still current and then attempts the swap. If that version is no longer current, the specification says the writer must retry the update based on the new current version.

What the specification deliberately does not do is prescribe one mechanism for making that swap atomic. It describes two schemes, and the difference is worth knowing because only one of them is current practice. For metastore tables, a pointer stored in a metastore or database is updated with a check-and-put that validates the version the write was based on is still current — the same optimistic pattern, executed by the catalog. For file system tables, the swap is an atomic rename of the new metadata file onto the well-known name for the next version, which works on file systems that support it, such as HDFS. The specification marks that second scheme as deprecated, to be removed in version 4, and states plainly that it is unsafe in object stores and local file systems.

So “how does a commit become atomic here” is a question about your catalog implementation rather than about the format, and the answer is not supplied by the storage layer’s capabilities. That S3 now offers conditional writes does not by itself mean the catalog you are running uses them. Check what your catalog does before assuming concurrent writers are safe.

One design detail is worth knowing because it decides what a retry costs. Sequence numbers are inherited from manifest metadata, which means a manifest can be written once and reused across commit retries; to change a sequence number for a retry, only the manifest list has to be rewritten. So a failed commit does not discard the data files or the manifests — it regenerates the manifest list and the metadata file. A retry is cheap in proportion to the work already done, which is why optimistic concurrency is viable here at all.

Delta Lake reaches the same guarantee with a different mechanism. Its log directory holds one newline-delimited JSON file per commit, named by a zero-padded version number, with periodic Parquet checkpoints so readers do not have to replay every entry. Atomicity comes from exclusivity on the version number: only one commit can claim a given version, so the writer that gets there first wins and the other retries. Readers reconstruct the current table state from the latest checkpoint plus the commits after it.

The shared guarantee is what matters for design: a reader sees one complete version or another, never a mixture, which is atomic publication combined with snapshot reads of the MVCC kind. The differences show up in operations — what a retry costs, how metadata grows, what maintenance is needed — so the team running a table should read the specification of the format it actually runs rather than a generic description of both.

Schema evolution, and why columns have IDs

A data file written last year has last year’s columns in last year’s order. If the table has since gained a column, lost one, and renamed a third, something has to reconcile the file with the current schema, and matching by name and position does not survive that.

Iceberg’s answer is to give every column a persistent identifier. The specification states that columns in data files are selected by field id, that the table schema’s names and order may change after a data file is written, and that projection must therefore be done using field ids. That one decision is what makes the supported operations safe: a struct, including the top-level schema, can evolve by deleting fields, adding new ones, renaming existing ones, reordering them, or promoting a primitive type through the allowed promotions.

Change	What happens underneath	What it costs
Add a column	A new field id is assigned; older files simply do not contain it, and defaults supply the value for existing rows	Nothing is rewritten
Rename a column	The field id is preserved and only the name changes	Nothing is rewritten; old files still resolve correctly
Drop a column	The field is removed from the schema; the bytes remain in existing files until those files are rewritten	Nothing immediate. If the column held sensitive data, the bytes are still there
Reorder columns	Only the schema’s ordering changes, since resolution is by id	Nothing is rewritten
Promote a type	Allowed for safe widenings only, and not for a field referenced by a partition field where the partition transform would produce a different value	Nothing is rewritten, but the restriction means some changes require a new table or a rewrite

The row worth pausing on is the third. Dropping a column is a metadata operation, so a schema change that makes a column disappear from queries does not remove it from storage. A team that drops a column to satisfy a privacy requirement has satisfied nothing until the files are rewritten and the old ones expired.

Deleting without rewriting

The same logic applies to rows, and for the same reason: the specification states that once written, data and metadata files are immutable until they are deleted. Removing one row out of a million cannot mean editing the file in place.

Iceberg records deletions in separate files, of two kinds. Position deletes mark a row deleted by data file path and row position within that file, encoded as a position delete file in format version 2 and as a deletion vector in version 3 and above. Equality deletes mark rows deleted by one or more column values, such as id = 5, which is what a streaming writer can produce without knowing where the row physically sits. Delta’s deletion vectors work on the same principle: mark rows as removed and leave the Parquet file in place.

This is cheap to write and not free to read. Every query against affected files now applies the delete information as it scans, and equality deletes are the more expensive kind because matching by value is broader work than skipping known positions. Deletes accumulate until a maintenance job merges them into rewritten data files, which is the same class of scheduled work as compaction and usually runs alongside it.

Three consequences follow, and they are the ones most often discovered late. Query performance on a heavily updated table degrades gradually as delete files pile up, in a way that looks like the engine getting slower. Erasure obligations are not met by a DELETE statement, because the bytes remain until a rewrite and an expiry remove them. And the storage bill includes the data files, the delete files, and every snapshot still retained — which is why retention settings deserve a deliberate decision rather than a default.

What the catalog adds

A table format defines a table given its location. It does not define what the table is called or where that location is, and the component that does is the catalog. This section describes the role generally rather than any product’s implementation.

• Naming and resolution. Turning sales.orders into the location of the current metadata, so that queries do not contain storage paths and data can be relocated without rewriting every query.
• The commit point. Where the format relies on the catalog to make the pointer swap atomic, the catalog becomes part of the correctness of every write, not merely a directory. That makes its availability a production concern.
• Operations spanning tables. Anything involving more than one table — creating several consistently, or renaming across a namespace — has to live above the table format, because the format’s guarantee stops at one table.
• Access control and discovery. Who may read which table, and how someone finds it without asking a colleague, which is what catalog coverage measures.

The practical question is how many catalogs an organization ends up with. Each engine tends to arrive with one, and a table registered in two catalogs has two answers to “what is the current version,” which is the one question the design was supposed to settle. The alternative — one catalog that several engines share — is what makes the open-format promise real, and it is also the decision most likely to constrain engine choice later. It is worth deciding deliberately rather than discovering after the second engine is in production.

Where the copies come from

Separation makes data easy to move, and easy movement produces copies that nobody decided to keep. Three paths account for most of them, and each has a different effect on cost and on what readers see.

Path	Why it happens	Cost	Consistency effect
Loading into an engine’s own storage	Performance, or a feature only available on internal tables	Storage twice, plus a pipeline to keep it current	How far the copy trails the source depends on the wait until the next load, the run time, and any retries or failed runs, so the lag has to be measured; two engines can now disagree
Cross-region replication	Latency for distant readers, or residency and continuity requirements	Storage per region plus transfer charges, which are often the surprise	The remote copy trails; a reader there sees an older snapshot
Caching, local or in-engine	Speed, usually automatic and invisible	Local disk or memory, and eviction that makes performance uneven	Depends entirely on what is cached: file bytes are safe because files are immutable, but a cached table version or file list can be stale

The caching row rewards a moment’s thought, because immutability changes the usual rules. Data files never change once written, so a cached data file cannot be wrong — it can only be unnecessary. What can go stale is the metadata: which snapshot is current, and which files belong to it. That is why engines that cache aggressively still refresh table metadata, and why a reader that has not refreshed keeps seeing a consistent older version rather than a corrupted new one. Iceberg’s own description of readers using the snapshot current at load time is exactly this behavior.

The general rule for copies is the one that applies to any replicated data: whichever copy is not authoritative needs a stated direction, a stated freshness, and a check that the two still agree, which is what reconciliation is for. Where a second engine reads the primary storage directly rather than holding a copy — the arrangement described in external tables and the lake/warehouse boundary — that problem does not arise, which is a substantial argument for paying the performance cost of reading in place.

Storage tiering deserves one caution in this context. Moving older objects to cheaper classes, as covered in storage tiering, lifecycle, and archival, works on objects, while the table format tracks which objects are current. A lifecycle rule that archives or deletes objects the table still references will produce a table that cannot be read, and the rule will look correct in isolation. Retention on the table and lifecycle on the bucket are two policies that have to be written together.

A short checklist for a separated stack

Question	If the answer is unclear
Which component decides what a table currently contains, and what happens if it is unavailable?	The catalog is a production dependency nobody is monitoring
What do two concurrent writers to the same table do?	Either silent corruption, or retry storms discovered under load
Can a second engine read these tables, including deletes and the current snapshot?	The format is open and the practical portability is not
Who runs compaction, delete merging, and snapshot expiry, and on what schedule?	Query time degrades gradually and storage grows without a cause anyone can name
What is the snapshot retention, and does it satisfy both time travel and erasure deadlines?	One of the two requirements is being missed silently
Do bucket lifecycle rules know which objects the table still references?	A table becomes unreadable through a storage policy that looked correct
Which copies of this data exist, and which one is authoritative?	Two engines answer the same question differently and nobody can say which is right

Questions to explore further

• For your most active table, how many snapshots are retained, and what would be lost if that number were halved?
• If you had to drop a column containing personal data tomorrow, what sequence of operations would actually remove the bytes, and how long would it take?
• How many catalogs does your organization have, and is any table registered in more than one?
• Which of your data copies exist because someone decided they should, and which exist because moving data was easy?

References

All sources were checked on September 15, 2026. Specification behavior differs by format version, and cloud provider consistency models have changed over time, so both are worth re-checking against the versions in use.

• Apache Iceberg, Table Spec
• Delta Lake, Transaction Log Protocol
• AWS, Amazon S3 User Guide: What is Amazon S3? (data consistency model)
• AWS, Amazon S3 User Guide: How to prevent object overwrites with conditional writes
• AWS, Amazon S3 User Guide: Renaming objects in directory buckets
• Apache Parquet Documentation, File Format