Table Format
A table format is a specification for treating a collection of data files as a single table. It records which files belong to the table at each version, what the table’s schema and partitioning are, and how a change becomes visible to readers all at once. Apache Iceberg and Delta Lake are two widely used examples.
It sits between two things that are easy to confuse with it. Below it is the file format, such as Parquet, which defines the layout inside one file and knows nothing about any other file. Above it is the catalog, which maps a table name to the table’s current metadata. The table format is the layer that answers the question neither can: which files are the table right now.
Why a directory of files is not a table
The older convention treated a directory as a table: every file under the path was part of it. That breaks as soon as anything changes.
- Partial writes are visible. A job that writes twenty new files and fails after twelve leaves twelve files that readers now include.
- Replacing data is not atomic. Rewriting a partition means deleting old files and adding new ones, and a reader in between sees neither version cleanly.
- Planning is slow. Finding the files means listing directories, which grows with the number of partitions and files.
- Schema lives elsewhere, or nowhere. Nothing in the directory says what the columns mean or how they changed over time.
Object storage makes these problems sharper, because a store such as Amazon S3 offers no way to change several objects in one atomic operation.
What a table format adds
The Iceberg specification states the core idea in its overview: the format tracks individual data files in a table instead of directories, so writers can create data files in place and add them to the table only through an explicit commit. Each commit that changes the table’s data produces a new snapshot, and every change produces new metadata that replaces the old with an atomic swap. The two are distinct: a schema or property change creates a new metadata version that can still point to the same current snapshot. From that design come the properties people actually use.
| Property | How it is achieved |
|---|---|
| Readers never see a half-finished write | New files are invisible until a commit makes a new snapshot current — an atomic publication — and readers keep using the snapshot they loaded |
| Concurrent writers do not corrupt the table | Only one commit can succeed against a given version; the other retries against the new one, using optimistic concurrency control |
| Fast planning | Metadata lists files with partition values and column statistics, so the engine can skip files without listing storage |
| Safe schema changes | The schema and its history are part of table metadata; Iceberg resolves columns by field id rather than by name |
| Row-level deletes and updates | Copy-on-write rewrites affected data files; merge-on-read can record changes separately, including delete files or deletion vectors. The format, engine, version, and operation determine which mechanism is available. Iceberg configuration exposes these as distinct write modes. |
| Time travel | Older snapshots remain readable until they are expired |
Delta Lake reaches similar guarantees with a different structure. Its protocol describes a transaction log of numbered commit files: writers first write new data files optimistically, then commit by adding a log entry that records which files to logically add and remove. Readers reconstruct a consistent snapshot from the log, and files no longer in the latest version are deleted later, after a retention period.
What it does not do
- It does not run queries. A table format is a specification plus libraries. An engine must implement it, and not every engine supports every feature or format version.
- It does not name or locate tables. That is the catalog’s job, and in Iceberg’s case the catalog often performs the atomic commit itself, which makes it part of write correctness.
- It does not clean up after itself. Old snapshots, small files, and accumulated delete files remain until maintenance jobs expire, compact, and rewrite them. Skipping that work is a risk rather than a certain slowdown: whether queries slow or bills grow depends on how many files and how much delete information the queries actually read, how much storage retained snapshots hold, and how it is billed. Give the work an owner, and decide when to run it from file counts, delete information per scanned file, scan range, measured latency, and retained storage.
- It does not guarantee portability by itself. A table is only as open as the set of engines that support the features it uses. A table written with a newer format version or feature can be unreadable to an older reader.
The practical takeaway is that choosing a table format is really choosing a set of commitments: which engines must read and write the tables, which catalog will perform commits, and who will run the maintenance. How the table format fits with object storage, catalogs, and engines is worked through in Storage, Compute, and Metadata: What Each Layer Actually Owns.
Snapshot reads also need a file-retention policy. Iceberg snapshot expiration can remove files no retained snapshot references, while Delta VACUUM uses a retention interval that must cover concurrent transactions and stream lag. A query holding a version number does not universally prevent cleanup from deleting its input. Before shortening retention, check the engine’s protection mechanisms and the longest work that still depends on an older version.
References
- Apache Iceberg, Table Spec
- Delta Lake, Delta Transaction Log Protocol
- Apache Iceberg, Configuration
- Delta Lake, Table utility commands: VACUUM retention and concurrent operations
- Apache Iceberg, Maintenance: snapshot expiration and orphan-file deletion
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.