File Compaction

File compaction rewrites several smaller data files into fewer, usually larger files while preserving the intended logical records. Compression reduces encoded byte size; compaction changes file organization. A rewrite may do both, but fewer files do not by themselves prove fewer bytes or unchanged content.

The reason many small files cost more is that each one carries a fixed charge, independent of its size. Spark makes the assumption explicit as a setting: spark.sql.files.openCostInBytes is “the estimated cost to open a file, measured by the number of bytes that could be scanned in the same time,” used when packing several files into one read partition, and its default is 4 MB. A 30 KB file is therefore planned as though it were 4 MB of scanning, and the documentation notes that over-estimating is deliberate. Add the scheduler’s per-task overhead and a listing that grows with file count, and a directory of thousands of tiny files is slow for reasons that have nothing to do with the volume of data in it.

Compaction trades lower per-file overhead against rewrite work and the layout that queries can exploit. Encoding and compression also depend on values, ordering, and block settings; enlarging a file alone does not guarantee an improvement. Smaller files permit finer file-level skipping when their value ranges are sufficiently distinct. Two small files containing 1 and 100, and 2 and 99, both span 50. Reorganizing the same values into files containing 1 and 2, and 99 and 100, lets min/max statistics exclude both for a query seeking 50. The file sizes did not change; the grouping did. A larger compacted file can also retain smaller row groups for skipping within it. Choose file size, row-group size, and ordering together, including the amount of data an update must rewrite.

Combining 1,440 files of 30,000 bytes into four files can preserve the same declared total of 43.2 decimal MB. A query can then need fewer object requests. Actual performance also depends on parallelism, row groups, range reads, shared bandwidth, decoding, and filtering. Four whole-file tasks cannot automatically use eight workers fully.

Prepare replacement files separately, validate record-level invariants, and publish the new snapshot through the table format’s commit protocol — the atomic publication step is what keeps readers from seeing a half-rewritten directory. Concurrent ingestion must not be lost. Leave the old files available until no retained snapshot or active reader needs them. Reading both old and replacement files through a broad listing can double-count records.

Choose sizes and frequency from measured workloads. And treat it as a standing responsibility with an owner rather than an occasional intervention. Where streaming ingestion or frequent small batches write, small files can keep accumulating unless the writer optimizes file sizes or the platform compacts automatically, so decide how often compaction is needed from the observed file-size distribution and from what automatic management already covers, not from a fixed calendar. Where the platform runs it, the owner’s job is to check that it is enabled, what it costs, and whether it works. Include rewrite compute, temporary duplicate storage, requests, retention, and recovery costs when estimating the benefit. Kafka log compaction, which retains keyed log state under another protocol, is a different concept.

See Designing an Object-Storage Data Lake for worked examples.

References

• Apache Iceberg maintenance
• Apache Spark Documentation, Performance Tuning
• DuckDB: Sorting on insert for fast selective queries