Field ID

A field ID is a permanent numeric identifier assigned to a column when it is added to a table’s schema. Table formats such as Apache Iceberg use it, rather than the column’s name or position, to find that column’s data inside data files. The name is what people see and may change; the ID is what the format uses and does not change.

Why names and positions are not enough

A table built on immutable files keeps data written under many past versions of its schema. A file written last year still has last year’s columns. Matching that file to today’s schema by name or by position breaks in ways that are easy to miss, because they produce wrong data rather than errors.

• Matching by position breaks when columns are reordered or one is dropped from the middle: every column after it lines up with the wrong data.
• Matching by name breaks when a name is reused. Suppose a column status is dropped and later a new, unrelated column is added with the same name. Old files still contain the old status values, and a name-based reader will present them as values of the new column.
• Renaming under name matching makes existing files appear to lack the column, so historical rows show nulls.

How field IDs solve it

The Iceberg specification states the rule directly: columns in data files are selected by field id, because the table schema’s names and order may change after a data file is written. The ID is written into each data file’s own schema — Parquet files store it as the field ID on the Parquet schema — so the file carries the IDs it was written with.

The specification’s example shows the effect. A file written with columns 1: a, 2: b, and 3: c is read with a current schema of 3: measurement, 2: name, 4: a. The reader returns the file’s column c as measurement, column b as name, and a column of nulls for a — because ID 4 is a new column that the file never contained, even though a column named a exists in it.

Three rules make this reliable:

• Adding a column always assigns a new ID. Table metadata records last-column-id, the highest ID ever assigned, so a new column never reuses an ID from a dropped one.
• Renaming changes the name, never the ID. Old files resolve to the renamed column without being rewritten.
• Missing IDs resolve predictably. If a file lacks an ID, the value comes from an identity partition value if there is one, then from a name mapping, then from the column’s default, and otherwise is null.

This is what lets Iceberg add, drop, rename, and reorder columns as metadata-only operations that leave existing files untouched. The same mechanism also resolves nested fields, list elements, and map keys and values, each of which has its own ID.

Delta Lake reaches the same place through column mapping. Its protocol assigns every column a physical name and a unique 32-bit integer ID, writes that ID into the Parquet field ID, and offers a mode in which readers resolve columns by that ID. Column mapping is a table feature that must be enabled; without it, Delta reads Parquet columns by their display names.

Where it needs care

• Files without IDs. Data files written before a table adopted a table format — for example, files imported from a Hive-style table — do not carry field IDs. Iceberg handles these with a name mapping property that assigns fallback IDs by name, and a mapping can list several names for one ID when files used different names. Getting that mapping wrong reintroduces the name-matching problems above. In Delta’s ID mode, the protocol says a reader must either refuse such a file or return nulls for its columns.
• Every writer must respect the IDs. A tool that writes Parquet files directly into a table’s location without the right IDs creates files the table cannot interpret correctly. Writes should go through a client that implements the format.
• A dropped column’s bytes remain. Removing a column from the schema stops it from being read, but the data stays in existing files until they are rewritten — a point that matters when the column held sensitive data.

Field IDs handle which column is which. Whether a column’s type can change is a separate question, covered by type promotion, and the general compatibility rules for changing schemas are covered in schema evolution and compatibility. What schema evolution actually costs underneath a lakehouse table is worked through in Storage, Compute, and Metadata: What Each Layer Actually Owns.

References: Apache Iceberg Table Spec, Column Projection; Apache Iceberg Table Spec, Schema Evolution; Delta Transaction Log Protocol, Column Mapping.