File Compaction

File compaction rewrites several smaller data files into fewer, usually larger files while preserving the intended logical records. Compression reduces encoded byte size; compaction changes file organization. A rewrite may do both, but fewer files do not by themselves prove fewer bytes or unchanged content.

Combining 1,440 files of 30,000 bytes into four files can preserve the same declared total of 43.2 decimal MB. A query can then need fewer object requests. Actual performance also depends on parallelism, row groups, range reads, shared bandwidth, decoding, and filtering. Four whole-file tasks cannot automatically use eight workers fully.

Prepare replacement files separately, validate record-level invariants, and publish the new snapshot through the table’s commit protocol. Concurrent ingestion must not be lost. Leave the old files available until no retained snapshot or active reader needs them. Reading both old and replacement files through a broad listing can double-count records.

Choose sizes and frequency from measured workloads. Include rewrite compute, temporary duplicate storage, requests, retention, and recovery costs when estimating the benefit. The main lab moves metadata and assumes equal sizes; it does not implement a Parquet rewrite. Kafka log compaction, which retains keyed log state under another protocol, is a different concept.

See Designing an Object-Storage Data Lake for worked examples.

Reference: Apache Iceberg maintenance.