Kafka Log Compaction

Kafka log compaction removes eligible superseded records for a key asynchronously. Remaining records retain their offsets and order, so offsets can have gaps. Several versions may coexist while cleaning is pending; a compacted topic is not immediately a table with exactly one row per key.

A keyed record with a null value is a tombstone used to express deletion. A state consumer removes that key when applying it. Tombstones can expire; a paused consumer that misses one may retain stale local state. Recovery can require rebuilding from a complete retained state into an empty target or reconciling with an authoritative snapshot.

Compaction preserves retained state rather than every historical event. Combining compact with delete also permits age or size based removal. Keep the key’s partition placement stable and check cleanup settings against bootstrap and outage duration. A Python dictionary of last values illustrates state application, not physical Kafka log cleaning.

Reference: Kafka fundamentals lab.