Open Table Formats: Snapshots and Catalogs

A directory of Parquet files looks like a table until two things happen at once. Consider a hypothetical payments dataset stored as files under one object-storage prefix. A nightly job replaces yesterday’s files: it deletes the old ones and writes new ones. A dashboard query starts in the middle of that job. It lists the prefix, finds some old files and some new files, and totals both. The totals are wrong, nothing failed, and no one can reconstruct afterward which files the query read, because the answer was a function of when the listing happened.

An open table format is the layer that fixes this. It decides which files belong to the table at each moment, records that decision in metadata, and changes the decision atomically. This article explains how that works in the three most visible implementations (Apache Iceberg, Delta Lake, and Apache Hudi): what transaction metadata each writes, what isolation a reader receives, how concurrent writers are reconciled, how schemas change without rewriting data, and what the catalog contributes. Version-dependent facts were read from the projects’ own documentation and release pages on October 3, 2026, and each is labeled with the version it came from.

The article assumes two earlier pieces. From Data Formats: Parquet, ORC, Avro, and Arrow, it uses the file footer, per-column statistics, and the fact that a Parquet file is immutable once written and describes only itself. From Data Modeling for Operational and Analytical Systems, it uses grain, keys, and the difference between keeping state and keeping history. The mechanisms below are explained here in full; the earlier articles supply only those background ideas.

What a table format adds to a pile of files

A table format is a specification, plus libraries that implement it, for treating a set of immutable data files as one table with versions. It sits between three neighbors that are easy to confuse with it.

Layer	Question it answers	Examples	What it does not know
File format	How are rows laid out inside one file?	Parquet, ORC, Avro	Which other files belong to the same table
Table format	Which files make up the table right now, and what was the table before?	Iceberg, Delta Lake, Hudi	How to execute a query; what the table is called
Catalog	Where is the current metadata for the table named payments.orders?	Hive Metastore, REST catalog, AWS Glue, JDBC database	What the files contain
Engine	How is a query planned and run over those files?	Spark, Flink, Trino	Whether a concurrent writer has committed

The Iceberg specification states the central idea compactly: the format tracks individual data files rather than directories, so writers can create files in place and add them to the table only through an explicit commit, and every change replaces the table’s metadata with an atomic swap (Iceberg table spec). The Delta protocol states the same two-phase shape in its own terms: writers first write new data files optimistically, and then commit by adding an entry to a log (Delta protocol). A file that has been written but not committed is invisible, which is why a failed job leaves garbage rather than corruption. The garbage still has to be collected, and Partitioning, Compaction, and Table Maintenance covers that maintenance.

The consequence worth stating before the details is that a lakehouse guarantee is the product of several parts. Object storage supplies durable immutable files. The table format supplies the commit protocol. The catalog often performs the one operation that must be atomic. The engine must implement the protocol correctly and support the format version in use. Remove any part and the guarantee weakens in a specific way, which the sections below point out as they go.

Transaction metadata: what a commit writes

Iceberg: a tree of immutable files

In Iceberg, every change to a table produces a new metadata file, and that metadata file points to a tree of other immutable files. The specification defines the pieces as follows.

• Table metadata file. A JSON file holding the schema (with a schema ID), the partition specs, the sort orders, custom properties, the list of snapshots, the named references (branches and tags), and a log of earlier metadata files.
• Snapshot. The state of the table at one point in time, identified by a snapshot ID, with an optional parent snapshot ID, a sequence number, a timestamp, a summary, and the location of a manifest list.
• Manifest list. One file per snapshot, listing that snapshot’s manifest files with summary statistics, including partition-value ranges and file counts, that let a planner skip whole manifests.
• Manifest. An immutable Avro file listing data files (or delete files) together with each file’s partition tuple and column metrics such as lower and upper bounds. Manifests are reused across snapshots so that slow-changing metadata is not rewritten.
• Data file. A Parquet, ORC, or Avro file containing rows.

metadata file v7 (JSON): schema, partition specs, refs {main -> snapshot 42}, snapshots [..., 41, 42]
  snapshot 42  (parent 41, sequence number 42)
    manifest list  snap-42.avro
      manifest m-17.avro  (partition range 2026-10-01 .. 2026-10-02, 120 files)
      manifest m-18.avro  (partition range 2026-10-03,            15 files)
        data file  part-0001.parquet  (partition 2026-10-03, per-column min/max, row count)
        data file  part-0002.parquet  ...

The diagram is a simplified illustration drawn for this article; real files carry more fields. Reading it from the top shows why planning is cheap. A reader loads one metadata file, follows the current snapshot to its manifest list, discards manifests whose partition summaries cannot match the query, and reads only the remaining manifests to find candidate data files. The specification’s stated goal is that planning uses a constant number of remote calls rather than a number that grows with the count of partitions or files (Iceberg table spec, Goals). Compare that with the directory model, where finding the files means listing storage, and the cost of the listing grows with the table.

Each successful commit also receives a sequence number. Sequence numbers matter for row-level deletes: a delete file generally applies only to data files with an older sequence number in the same partition, so the order of commits determines which rows a delete removes. When a commit must be retried, the writer reassigns the sequence number and writes a new manifest list, but the manifests it already wrote can be reused, because their entries inherit the sequence number from the manifest list at read time. That detail is the reason a retry is cheap in the common case.

Delta Lake: a log of numbered commits

Delta Lake stores table state as an ordered log. Each commit is a JSON file in the table’s _delta_log directory whose name is a zero-padded version number, and each file holds the actions that commit performed: files added, files removed, metadata changes, protocol changes, and transaction identifiers for streaming writers. The protocol describes a table as having a single serial history of atomic versions named by contiguous, monotonically increasing integers (Delta protocol).

_delta_log/
  00000000000000000042.json              <- commit: add part-0007.parquet, remove part-0003.parquet
  00000000000000000042.checkpoint.parquet <- full table state as of version 42
  00000000000000000043.json
  00000000000000000044.json
  _last_checkpoint

A reader reconstructs a snapshot by finding the newest checkpoint and replaying the commit files after it. A checkpoint is a Parquet file holding the complete replayed state up to its version: the protocol and metadata, the files that have been added and not removed, and recently removed files that have not yet expired. The protocol example uses a layout like the one above, where a checkpoint at version 42 covers everything earlier and versions 43 onward are replayed on top. Checkpoints bound the replay cost. Without them, planning cost would grow with the number of commits since the table was created, which is why the protocol encourages writers to checkpoint reasonably often.

Hudi: a timeline of instants

Apache Hudi records every action on a table, including writes, compactions, cleaning, and clustering, on a timeline. An action has a requested instant time and a completed instant time, and the documentation says a commit is published atomically to the timeline (Hudi timeline). Hudi’s table services (compaction, clustering, cleaning) are first-class timeline actions, which is a different emphasis from Iceberg and Delta, where compaction is an ordinary commit that replaces files. The comparison table near the end lays the three side by side.

What the three structures have in common

The structures differ in shape, but each answers the same four questions. Which files are in the table at version V? What is the schema at V? What is the next version, and who is allowed to create it? And which older versions can still be read? The first two are metadata content. The third is the commit protocol, covered next. The fourth is the retention policy, which this article treats as a retention setting and the next article treats as an operation.

There is a cost to keeping versions. Every commit adds a metadata file, so a table that commits every minute accumulates 1,440 snapshots a day. Iceberg tracks previous metadata files in the metadata log up to write.metadata.previous-versions-max (default 100) and deletes none of them unless write.metadata.delete-after-commit.enabled is turned on; the maintenance documentation points out that metadata files that fall out of the tracked log become orphans that only an orphan-file cleanup removes (Iceberg maintenance). Treating metadata as free is a common way for a streaming table to degrade slowly.

The atomic step: swapping a pointer

Everything above is built from immutable files, and creating an immutable file is harmless until something points to it. The single operation that must be atomic is the one that makes new metadata current. The Iceberg specification gives the protocol for the metastore-backed case in six steps: create new table metadata based on the current metadata, write it to a uniquely named file, ask the metastore to swap the table’s metadata pointer from the old location to the new one, treat a successful swap as a successful commit, and treat a failed swap as proof that another writer created the next version first, in which case start again from step 1 (Iceberg table spec, Metastore Tables). The swap is a check-and-put: it succeeds only if the version the writer started from is still current.

This is optimistic concurrency control applied to a whole table. The writer does the expensive work (writing data files and metadata) assuming no one else will interfere, then validates and commits. Under low contention the cost is one small check. Under high contention the cost shows up as retries, which the concurrency section quantifies.

The specification also documents an earlier scheme in which the commit is a rename of the new metadata file to a well-known name, and it marks that scheme as deprecated, to be removed in version 4, because it is unsafe in object stores and on local file systems. That deprecation is the practical reason the catalog matters for correctness. Object stores traditionally had no atomic rename, so the pointer swap moved into a service that can do a conditional update.

The Iceberg REST catalog API makes the check explicit. A commit to a table (updateTable) carries requirements and updates. Requirements are assertions validated before the changes are applied; the OpenAPI description gives assert-ref-snapshot-id, which checks that a named reference points to a certain snapshot ID, as the example. A failed requirement returns HTTP 409 with CommitFailedException, which the description says the client may retry. A server-side error such as a 500, 502, or 504 is documented as leaving the commit state unknown (Iceberg REST catalog OpenAPI). The same specification includes a multi-table commit endpoint, /v1/{prefix}/transactions/commit, which commits updates to several tables as one atomic operation.

The unknown-state outcome deserves attention because it is the failure that retry logic gets wrong. If a connection drops after the request was sent, the writer cannot tell whether the commit happened. Blindly retrying could apply an append twice. Iceberg’s table properties include commit.status-check.num-retries (default 3), with minimum and maximum waits of 1 second and 1 minute, which control how many times a writer checks whether its commit actually landed before giving up and reporting an unknown commit state (Iceberg configuration). The correct response to “unknown” is to check the table, not to repeat the write.

Delta’s analog is the creation of the next numbered commit file. The documentation says multiple writers across clusters can modify a table and see a serial order of writes “for supported storage systems” (Delta concurrency control), and the protocol additionally defines catalog-managed tables in which a catalog participates in ratifying commits and file-system-level access is not supported. The qualification matters: the guarantee is a property of the table format together with a storage or catalog layer that can make the commit step atomic. Hudi takes a different route: it relies on a lock provider so that writers hold a lock only briefly around the commit, and its documentation lists a storage-based lock provider that uses conditional writes on a single lock file in the table’s own storage (Hudi concurrency control).

Snapshot isolation: what a reader gets

The Iceberg overview says it directly: readers use the snapshot that was current when they loaded the table metadata and are not affected by later changes until they refresh and pick up a new metadata location. Delta documentation says the same of Spark jobs: a reader continues to see the consistent snapshot the job started with, even if the table is modified during the job. Both follow from immutability. Data files and manifests are never edited, so a reader holding a pointer to a snapshot is reading a fixed set of files. This is the table-level version of the MVCC idea used inside relational databases, and the word snapshot here means a defined table state, which is different from the periodic snapshot copies discussed in the data modeling article.

Three consequences follow, and each has a limit.

• Readers never block writers, and writers never block readers. Neither takes locks. The limit is cleanup: a reader pinned to an old snapshot still needs that snapshot’s files. If maintenance deletes the files while a long query is running, the query fails or reads incomplete data. Delta’s documentation recommends a vacuum retention of at least 7 days, longer than the longest running transaction and the longest stream lag, for this reason (Delta utility commands). Iceberg’s expiration deletes files only when no retained snapshot references them, but a query that outlives its snapshot’s retention has the same problem.
• Time travel is the same mechanism with a different starting pointer. Reading an older snapshot ID or timestamp reads an older file set. Iceberg also supports named references: a tag labels one snapshot, and a branch is a mutable reference that advances by committing new snapshots, with its own retention settings.
• A snapshot is consistent within one table, not across tables. Two tables read at different moments can disagree. A multi-table commit endpoint exists in the REST specification, but a query that joins two tables still needs both read at compatible versions, which the format does not arrange by itself.

Snapshot isolation versus serializable writes

Reading from a snapshot does not by itself settle what happens when two writers act on overlapping data. The Iceberg specification lists serializable isolation as a goal and then adds that the conditions a write must satisfy in order to commit determine the isolation level, and that writers can select what to validate (Iceberg table spec, Optimistic Concurrency). In the configuration, row-level commands (write.delete.isolation-level, write.update.isolation-level, write.merge.isolation-level) each accept serializable or snapshot and default to serializable (Iceberg configuration).

The distinction is the standard one from database theory, summarized in transaction isolation. Under snapshot isolation a transaction works from a fixed snapshot, and a commit conflicts only if another transaction changed the same data. Under serializable isolation the outcome must be equivalent to some serial order, so a transaction that made a decision based on rows it read must also fail if a concurrent commit added a row that would have changed that decision (a phantom, or a case of write skew). Here is a table-level illustration. Suppose a job deletes every payments row with status = 'void' while another job concurrently appends a new void row. A check limited to files the delete rewrote will not notice the new file. A check that the data matching the delete’s filter has not changed will. Which exact checks the Spark implementation performs at each setting was not traced, so treat that description as the general meaning of the two names and confirm the behavior of your engine and version before depending on it.

Delta Lake describes its writes as serializable ACID with a serial order, and its history table reports an isolationLevel for each operation. Hudi lists snapshot isolation between writers, table services, and readers, optimistic concurrency control between writers, and multiversion concurrency control between writers and table services (Hudi concurrency control). Names alone do not make the guarantees equivalent. Compare them by asking which concurrent operations each system can reject.

A toy model of the commit protocol

The following program is a deliberately small model, written for this article, of the pointer-swap commit described above. It is not Iceberg or Delta code. It models a table as an immutable snapshot (a set of file names) and a catalog that holds one pointer and swaps it only if the caller’s base is still current. It then shows four situations: a reader pinned to an old snapshot, two racing appends, a compaction racing an append, and a compaction racing a rewrite of one of its input files. Output is shown from running this exact script with Python 3.12 and only the standard library.

"""Toy model of an optimistic table commit (illustrative; not Iceberg or Delta code)."""
import threading

class Catalog:
    """Holds one pointer per table and swaps it only if the caller's base is still current."""
    def __init__(self, initial):
        self._lock = threading.Lock()
        self.current = initial            # immutable metadata dict
    def load(self):
        return self.current
    def swap(self, base, new):            # check-and-put
        with self._lock:
            if self.current is base:
                self.current = new
                return True
            return False

def snapshot(parent, files):
    return {"id": (parent["id"] + 1) if parent else 1, "files": frozenset(files)}

class ConflictError(Exception): pass

def append(cat, new_files, hook=None, log=None):
    attempts = 0
    while True:
        attempts += 1
        base = cat.load()
        if hook: hook(attempts)           # lets the demo interleave another writer
        new = snapshot(base, base["files"] | set(new_files))   # append: no validation needed
        if cat.swap(base, new):
            return new, attempts

def rewrite(cat, remove, add, hook=None):
    base = cat.load()
    if hook: hook()
    while True:
        # replace operations must verify the files they remove are still in the table
        cur = cat.load()
        if not set(remove) <= cur["files"]:
            raise ConflictError(f"files already removed: {sorted(set(remove) - cur['files'])}")
        new = snapshot(cur, (cur["files"] - set(remove)) | set(add))
        if cat.swap(cur, new):
            return new

# 1. A reader keeps the snapshot it loaded
cat = Catalog(snapshot(None, {"a.parquet", "b.parquet"}))
reader_view = cat.load()
append(cat, {"c.parquet"})
print("reader still sees   :", sorted(reader_view["files"]), "snapshot", reader_view["id"])
print("new reader sees     :", sorted(cat.load()["files"]), "snapshot", cat.load()["id"])

# 2. Two appends race from the same base: loser retries on the new base and succeeds
cat = Catalog(snapshot(None, {"a.parquet"}))
def other(attempt):
    if attempt == 1:
        append(cat, {"x.parquet"})
_, n = append(cat, {"y.parquet"}, hook=other)
print("append vs append    :", sorted(cat.load()["files"]), "attempts for second writer =", n)

# 3. Compaction races with an append: compaction's removed files are untouched, so it can retry
cat = Catalog(snapshot(None, {"s1", "s2", "s3"}))
rewrite(cat, {"s1", "s2", "s3"}, {"big"}, hook=lambda: append(cat, {"s4"}))
print("compact vs append   :", sorted(cat.load()["files"]))

# 4. Compaction races with a delete that rewrote one of its input files: conflict
cat = Catalog(snapshot(None, {"s1", "s2", "s3"}))
try:
    rewrite(cat, {"s1", "s2", "s3"}, {"big"},
            hook=lambda: rewrite(cat, {"s2"}, {"s2_without_deleted_rows"}))
except ConflictError as e:
    print("compact vs rewrite  : ConflictError,", e)
print("final files         :", sorted(cat.load()["files"]), "snapshot", cat.load()["id"])

reader still sees   : ['a.parquet', 'b.parquet'] snapshot 1
new reader sees     : ['a.parquet', 'b.parquet', 'c.parquet'] snapshot 2
append vs append    : ['a.parquet', 'x.parquet', 'y.parquet'] attempts for second writer = 2
compact vs append   : ['big', 's4']
compact vs rewrite  : ConflictError, files already removed: ['s2']
final files         : ['s1', 's2_without_deleted_rows', 's3'] snapshot 2

Read the output line by line. The first two lines are snapshot isolation: the reader’s object is unchanged by the append, while a reader that loads the table afterward sees snapshot 2. The third line shows the optimistic loop. The second writer loaded snapshot 1, another writer committed first, the swap failed because the base was no longer current, and the retry rebuilt its change on the new base, so both files appear and the second writer needed two attempts. The append needed no validation, matching the specification’s statement that append operations have no requirements and can always be applied.

The fourth line shows a replace operation whose precondition is still satisfied. The compaction removes s1, s2, and s3. Since the concurrent append added only s4, none of the files it removes has changed, and the compaction commits on top of the new state. The last two lines show the case that fails. A concurrent rewrite replaced s2 with a version that has rows removed. The compaction’s input is now partly gone, and committing would silently resurrect the deleted rows in big, so it raises an error. The specification requires exactly this validation: replace operations, with compaction as the named example, must verify that the files they delete are still in the table.

The model leaves out most of what makes real systems hard: bounded retries, backoff, partial progress, the unknown-commit state, delete files, and the separation of data-file writes from metadata writes. Its job is only to make the shape of the protocol concrete. The conflict rules it exercises are the ones the next section tabulates for real systems.

Concurrency: who can conflict with whom

Iceberg’s specification summarizes retry rules by operation type. Append operations have no requirements and can always be applied to a newer version. Replace operations (format changes and compactions) must verify that the files being deleted are still in the table. Delete operations must verify that the specific files to delete are still present, while expression-based deletes such as where timestamp < X can always be applied. Schema updates and partition-spec changes must validate that the schema has not changed between the base version and the current version (Iceberg table spec, Commit Conflict Resolution and Retry).

Delta’s documentation gives the equivalent matrix with a compaction defined as a file-compaction operation that sets dataChange to false (Delta concurrency control).

Operation pair (Delta Lake)	INSERT	UPDATE, DELETE, MERGE INTO	COMPACTION
INSERT	Cannot conflict	Can conflict	Cannot conflict
UPDATE, DELETE, MERGE INTO	Can conflict	Can conflict	Can conflict
COMPACTION	Cannot conflict	Can conflict	Can conflict

“Can conflict” is not “will conflict”. The documentation states that whether two operations conflict depends on whether they operate on the same set of files, and that partitioning by the columns used in the commands’ conditions makes the file sets disjoint. Its example: UPDATE ... WHERE date > '2010-01-01' and DELETE ... WHERE date < '2010-01-01' conflict if the table is not partitioned by date, because both can modify the same files. It adds that partitioning by a high-cardinality column can create other problems through a large number of subdirectories. Partition design is therefore a concurrency decision as well as a read-performance decision, a point the partitioning article develops.

Delta’s documentation also names the exceptions a failed commit raises, which are useful diagnostics: ConcurrentAppendException when a concurrent operation added files to a partition your operation read, ConcurrentDeleteReadException when a concurrent operation deleted a file your operation read, ConcurrentDeleteDeleteException when two operations delete the same file (two compactions over the same files, for example), MetadataChangedException for concurrent schema or table-property changes, and ProtocolChangedException when the table’s protocol was upgraded or multiple writers created the table at once. The documentation’s merge example shows the common fix for the first: add the partition values to the merge condition so the operation provably reads only its own partition.

Hudi offers three concurrency modes, in its own words: optimistic concurrency control between writers for relational-style semantics, multiversion concurrency control between writers and table services, and non-blocking concurrency control between writers to give streaming semantics and avoid live-lock and starvation between writers. Multi-writer setups need a lock provider; a single process that performs both writing and its own table services can use an in-process lock provider and avoid distributed lock infrastructure (Hudi concurrency control).

Retry cost under contention

Optimistic concurrency is cheap when conflicts are rare and expensive when they are common, and a simple calculation shows how quickly. Suppose k writers start from the same base and each tries to commit. Exactly one succeeds in the first round, k – 1 retry, one of those succeeds in the second round, and so on. In the worst case, where every loser retries immediately and all remaining writers collide each round, the total number of commit attempts is \(k + (k-1) + \dots + 1 = k(k+1)/2\). For 20 writers that is 210 attempts to produce 20 commits, and the writer that loses the most rounds makes 20 attempts. Iceberg’s default of 4 retries (commit.retry.num-retries) would fail that writer, since it would need 19 retries. This is an upper bound under assumptions that backoff and jitter relax (the defaults include waits between 100 ms and 60 s and a 30-minute total timeout), and many appends need no revalidation, so each retry is cheap. Even so, the quadratic shape explains why a hot table fed by many independent writers is a design problem to solve, not a setting to tune.

Three design responses are common. Funnel writes through fewer committers, for example one streaming job that commits every minute rather than hundreds of tasks committing independently. Keep conflicting operations in disjoint partitions or filter ranges so that validation passes. Schedule compaction so that it does not rewrite files that row-level jobs are also changing; Iceberg’s compaction procedure exposes partial-progress options for this, which the next article describes. Each response trades throughput or freshness for fewer conflicts, and none of them removes the need to handle a commit failure in the job.

Schema evolution without rewriting data

A Parquet file’s schema is fixed when the file is written. The earlier Parquet article showed that a reader can combine files with different schemas only if it has rules for matching columns. Name-based matching and position-based matching both fail on common changes. Rename a column and name-based readers treat the old files as missing the new name. Drop a column and later add a different one with the same name, and name-based readers can quietly attach old values to the new column. Drop a column in the middle and position-based readers shift every later value.

Iceberg resolves this by assigning each column a unique ID when it is created and recording the ID in both the table schema and the data files’ column metadata, so the mapping is by ID. The documentation lists five changes, all metadata-only: add, drop, rename, update (widen a type), and reorder, including inside nested structs, and says it guarantees that these are independent and free of side effects. Added columns never read existing values from another column, dropping a column does not change any other column, and reordering does not change which values belong to which name (Iceberg evolution). It then states the failure of the alternatives: formats that track columns by name can un-delete a column when a name is reused, and formats that track by position cannot drop a column without changing the names used for the remaining ones.

Type changes are restricted to safe widenings. The specification’s table for versions 1 and 2 allows int to long, float to double, and decimal(P, S) to decimal(P', S) where P’ > P, widening precision only. Version 3 additionally allows date to timestamp and timestamp_ns (but not to the zoned variants) and treats unknown as promotable to any type (Iceberg table spec, Schema Evolution). Narrowing is not allowed, because it could lose information for rows already written. The specification also records a subtle consequence: the Avro manifests do not store the type of lower and upper bounds, and promotion does not rewrite them, so a float column promoted to double still has 4-byte bounds in older manifests, and readers must infer the original type from the bounds’ length. Schema evolution is easy to describe and has implementation corners like this one.

Delta Lake reaches the same goal differently. Its column mapping feature gives each column a physical name and a unique integer ID stored in the column’s metadata, so renaming or dropping a column does not require rewriting data files. Mapping works in name or id mode and requires reader and writer protocol versions that support the feature (Delta protocol, Column Mapping). The point is shared: the logical name in the schema is decoupled from the identifier the files are matched by.

Partition evolution

Iceberg treats partitioning as metadata too. A partition spec is versioned in the table metadata; changing it adds a new spec, and old data keeps the layout it was written with. The documentation’s example is a table partitioned by month for 2008 and by day from 2009, with both layouts coexisting. A query is planned once per spec, using the filter translated for that spec, and this is called split planning. Because Iceberg’s hidden partitioning derives partition values from column values through transforms such as day(ts) or bucket(16, id), queries filter on the original column and do not need to know the layout (Iceberg partitioning). That decoupling is what makes it safe to change the layout later. The decision of which layout to choose is the subject of Partitioning, Compaction, and Table Maintenance.

What schema evolution does not solve

• Meaning. A column can keep its ID and type while its unit changes from dollars to cents. Structural evolution cannot detect that. The schema evolution and compatibility glossary entry separates structural compatibility from semantic change, and the semantic part needs a written contract and downstream notification.
• Readers. A table written with a newer format version or feature can be unreadable to an older reader. Iceberg’s specification increments the format version when a new feature breaks forward compatibility, and it notes that tables may continue to be written with an older version to avoid features not yet implemented by the processing engines in use.
• Concurrency. A schema change is itself a commit. In both Iceberg and Delta it conflicts with concurrent operations that depend on the old schema, so run it in a quiet window or expect retries.

Catalogs: who holds the pointer

A table format defines what the metadata looks like, and the catalog maps a table name to the location of its current metadata file. Spark’s Iceberg configuration shows the common choices. A single Spark catalog implementation supports Hive, Hadoop, REST, Glue, JDBC, and Nessie catalogs, selected by a type property (Iceberg Spark configuration). A Hive Metastore keeps table definitions, locations, and partition information in a metadata database; the Iceberg Hive catalog uses it to store the metadata pointer. A data catalog in the broader governance sense also describes ownership and meaning, which a technical metastore does not.

The catalog role is more than a lookup in two ways, and each has operational consequences.

1. The catalog often performs the atomic commit. In the metastore scheme the pointer swap is the catalog’s operation, and in the REST scheme the server validates requirements. A catalog that cannot perform a conditional update cannot safely coordinate concurrent writers. The implication for design is that the catalog must be at least as available and as durable as the tables whose commits it arbitrates. If the catalog is down, readers that need to resolve the table name fail, and writers cannot commit, even though every data file is intact.
2. The catalog is an access-control and credential boundary. Since all engines that go through a catalog share one view of the table’s current state, it is a natural place to enforce who may read or commit. The 1.12.0 release notes list a change adding finer-grained read restrictions to the loadTable response in the REST specification, which can be read as a sign that the catalog is expected to carry part of the access decision. That is an interpretation of one release note; catalogs differ in what they enforce, so check what yours does.

One failure follows from the pointer model and is an inference from the protocol rather than a documented incident. If the same table is registered in two catalogs, each catalog holds its own pointer, and a commit through one is invisible to the other. Both can accept check-and-put operations that succeed against their own copy, and the table’s history forks. The protocol cannot detect this because the check is against a single pointer. Register each table in one catalog and make every writer go through it. A related fact from the Iceberg Spark documentation: the Hive-based Iceberg catalog loads only Iceberg tables, so non-Iceberg tables in the same metastore need a session catalog.

Comparing the implementations, with verified versions

The versions below were checked on October 3, 2026, from the projects’ release pages and documentation. Release information changes, so treat this table as a dated snapshot of that day.

Item	Apache Iceberg	Delta Lake	Apache Hudi
Latest release at check time	1.12.0, listed as released September 29, 2026 on the Iceberg releases page (GitHub release published September 30 UTC)	v4.4.0, published on GitHub August 20, 2026. The documentation’s compatibility table still lists 4.3.x as its newest row	release-1.2.1, published on GitHub September 24, 2026
Format or protocol version state	Spec versions 1, 2, and 3 are adopted; version 4 is under active development and not formally adopted	Protocol described by reader and writer versions plus named table features	Not examined for this article
Where table state lives	Metadata JSON, snapshot, manifest list, manifests	Numbered JSON commit files in _delta_log plus Parquet checkpoints	Timeline of requested and completed instants
Commit step	Atomic swap of the metadata pointer (catalog check-and-put or REST requirements)	Creation of the next numbered log entry on storage that supports it, or catalog-ratified commits for catalog-managed tables	Atomic publication to the timeline, with lock providers coordinating multiple writers
Concurrency control	Optimistic; validation rules differ by operation; configurable isolation level for row-level commands	Optimistic; conflict matrix by operation type; named conflict exceptions	OCC between writers, MVCC between writers and services, optional NBCC
Schema change mechanism	Column IDs; metadata-only add, drop, rename, widen, reorder	Column mapping with physical names and IDs	Not examined for this article
Documented maintenance operations	Expire snapshots, remove orphan files, rewrite data files, rewrite manifests	OPTIMIZE (compaction and Z-ordering), VACUUM	Compaction, clustering, and cleaning appear as table-service actions on the timeline

Two comments on the table. First, the empty cells are honest. Reading the Hudi schema-evolution rules and the format-version state was outside the sources examined, and an unverified cell would not help a reader. Second, the table compares mechanisms, not speed or cost. No benchmark was run for this article, and none of the sources read gives a result that would support a claim that one project is faster. Throughput depends on file sizes, partition layout, engine, catalog latency, and write pattern, so compare candidates by running your own write pattern against each.

The Iceberg 1.12.0 release notes also illustrate how a fast-moving format treats compatibility. They list removal of support for Spark 3.4 and for Flink 2.0, removal of several deprecated APIs, and ongoing work on a version 4 metadata layout, including relative paths so that tables can be relocated without rewriting metadata. For an operator, the lesson is to pin the engine and library versions that read and write a table, to read release notes before upgrading, and to upgrade readers before writers when a new table format version is introduced.

Choosing: criteria that decide the question

A defensible choice comes from constraints rather than from a feature checklist. These are the questions that separate the options in practice.

1. Which engines must read and write? A format is only as open as the engines that implement the features you use. List the engines, and for each, the format version and feature set it supports at the version you run.
2. Which catalog will arbitrate commits, and who operates it? A catalog that cannot do a conditional update, or that is less available than the data it governs, undermines the guarantee.
3. What is the write pattern? Many independent appenders, a few large overwrites, and frequent row-level updates stress concurrency control in different ways. The conflict matrices above show which combinations can conflict.
4. What is the history requirement? Time travel and rollback depend on retained snapshots, and retained snapshots cost storage and metadata. Set retention from the recovery and audit requirement, not from the default.
5. Who owns maintenance? Compaction, expiry, and orphan cleanup do not run themselves, and a missed schedule shows up as slow queries or a storage bill. This is the subject of Partitioning, Compaction, and Table Maintenance.

Not every table needs this layer. A small, append-only, single-writer dataset consumed by one engine may be adequately served by a partitioned directory and a publication convention. The table format earns its complexity when there are concurrent writers, row-level changes, schema changes over time, or readers who must see consistent versions.

Failure modes and where to look

Symptom	Likely cause	Where to look
Writer fails with a conflict or CommitFailedException	Another commit changed what the writer depended on	Which operation types ran concurrently; whether the conflicting files overlap by partition
Job reports unknown commit state	Connection lost after the commit request	Check the table’s current snapshot or version before any retry
Long query fails on missing files	Snapshot expired or vacuum retention shorter than the query	Retention settings versus longest query and stream lag
Planning gets slower over weeks	Many snapshots, small files, or untidy manifests	File count, manifest count, and metadata log size
Two engines show different table state	Table registered in two catalogs, or an engine caching metadata	Which catalog holds the authoritative pointer
Old engine cannot read a table	Table uses a newer format version or feature	Table format version and the engine’s supported feature list

The common thread is that nearly every table-format failure is a metadata problem: a pointer that moved, a version that expired, or a feature that a reader does not know. Debugging starts by reading the metadata, not the data files.

References

• Apache Iceberg, Table Spec (overview, optimistic concurrency, snapshots, schema and partition evolution, commit conflict resolution and retry, metastore tables), read October 3, 2026.
• Apache Iceberg, Configuration (table properties for isolation level, commit retries, metadata retention), read October 3, 2026.
• Apache Iceberg, Evolution, read October 3, 2026.
• Apache Iceberg, Partitioning (hidden partitioning), read October 3, 2026.
• Apache Iceberg, Maintenance, read October 3, 2026.
• Apache Iceberg, Spark Configuration (catalog types), read October 3, 2026.
• Apache Iceberg, Releases (1.12.0 release notes), read October 3, 2026.
• Apache Iceberg, REST catalog OpenAPI specification (updateTable, requirements, multi-table commit), read October 3, 2026.
• Delta Lake, Delta Transaction Log Protocol (log entries, checkpoints, column mapping, catalog-managed tables), read October 3, 2026.
• Delta Lake documentation, Concurrency control, read October 3, 2026.
• Delta Lake documentation, Utility commands (VACUUM retention), read October 3, 2026.
• Delta Lake documentation, Releases (compatibility table) and Delta Lake GitHub releases, read October 3, 2026.
• Apache Hudi, Concurrency Control (version 1.2.1 documentation), read October 3, 2026.
• Apache Hudi, Timeline, read October 3, 2026.
• Apache Hudi GitHub releases, read October 3, 2026.