Log Compaction

Log compaction is a retention policy that keeps, in Kafka’s words, “at least the last known value for each message key within the log of data for a single topic partition.” Older records for a key are eventually removed; the most recent one stays, however old it is. Descriptions follow Apache Kafka 4.3.X documentation, checked in September 2026.

Despite the similar name, this is unrelated to compacting small files in a data lake — see file compaction for that.

Two ways a log forgets

Kafka’s documentation motivates the second with an example: a topic of user email addresses keyed by user ID, where the same user updates the address several times. Time-based retention eventually drops all of it. Compaction keeps the final value, so — in the documentation’s phrasing — the log contains a full snapshot of the final value for every key, not just the keys that changed recently.

What it is for

The documented use cases are state-shaped: restoring state after an application crash or system failure, and reloading caches after a restart. A service that keeps an in-memory view of “current price per product” can rebuild it by reading a compacted topic from the beginning, without querying the owning system — subject to one timing condition covered below — which is what makes such a topic usable as a distributed source of current state rather than only as a transport.

Deletions fit into the same model through a record with a key and a null value, a tombstone, which marks the key as removed so that consumers rebuilding state drop it too.

That last part carries a time condition, and it is the difference between a rebuilt cache that is correct and one that quietly holds deleted keys. Kafka’s guarantee is stated with the qualifier attached: a consumer reading from the start “will see at least the final state of all records in the order they were written,” and “all delete markers for deleted records will be seen, provided the consumer reaches the head of the log in a time period less than the topic’s delete.retention.ms setting (the default is 24 hours).” The reason is that removal of the markers happens while you are reading — “it is possible for a consumer to miss delete markers if it lags by more than delete.retention.ms.”

Two rebuild situations come out of that differently.

• Rebuilding from empty. A scan that finishes within the marker retention window ends with the deleted keys absent, which is what you want. A slow scan — a large topic, a throttled consumer, a restart partway — can pass a key’s marker after it has been cleaned and end up holding a value that was deleted. The state is wrong and nothing reports it.
• Resuming an existing state. A cache that was offline longer than the marker retention may never see the deletions that happened while it was away, because the markers are gone and the keys are simply absent from the log. Resuming from a stored position is not sufficient here; the safe move is to rebuild from the beginning.

So the operational question is how long a full scan takes relative to the topic’s marker retention, and the settings worth reading before relying on any of this are delete.retention.ms, plus min.compaction.lag.ms and max.compaction.lag.ms, which bound when a record becomes eligible for compaction. And if the topic’s policy is compact,delete rather than compaction alone, time-based deletion applies as well — the latest value for a key is not exempt from it, which removes the “full snapshot of the final value for every key” property the use cases above assume.

What it deliberately loses

• History. Compaction preserves the current value, not the sequence of changes that produced it. A topic that must answer “what was the price in March” needs a different retention policy or a separate history table.
• Timing guarantees about removal. Compaction is a background process, so a superseded value may remain readable for a while. Consumers must tolerate seeing an old value followed by a newer one.
• Meaning for unkeyed records. Compaction works per key; records written without one have nothing to compact against.

The clean way to decide: if consumers need to reconstruct the present, compact; if they need to replay what happened, keep the events and set the window from the longest outage you intend to survive. How this fits with partitions, replication, and consumer groups is worked through in Kafka as a Log.

References: Apache Kafka documentation, Design (Log Compaction); Apache Kafka documentation, Broker Configs; Apache Kafka 4.1 Documentation, Design.