Kafka as a Log: Partitions, Replicas, and What It Still Does Not Guarantee

Teams adopt Kafka expecting a message queue and get something with different properties: events are not removed when they are read, order holds in one place and not another, and the consumer — not the broker — decides what has been processed. Most of the surprises that follow come from one design decision, which is that a topic is a set of append-only logs rather than a queue of pending work.

This article works through that model: how data is stored and ordered, how replication decides what “written” means, how consumers track their own position, and what remains your responsibility afterwards. Descriptions and defaults follow Apache Kafka 4.3.X documentation, checked in September 2026.

Topics, partitions, offsets

A topic is split into partitions spread across brokers, and each partition is an ordered, append-only log. Every record in a partition has an offset: its position in that log. Kafka’s introduction states the ordering rule precisely — “events with the same event key… are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition’s events in exactly the same order as they were written.”

Read that sentence for what it does not say. There is no ordering guarantee across partitions. Two events written a millisecond apart to different partitions can be read in either order.

Be precise about where the guarantee lives, because the shorthand “order is per key” hides the mechanism. The ordering is a property of the topic partition: every record in one partition is ordered against every other record in it, keyed or not. A key matters because the default partitioner sends the same key to the same partition, so records sharing a key land in one ordered sequence. That makes the choice of key a correctness decision rather than a routing detail — and it also means the partition, not the key, is the unit of ordering, parallelism, and replication.

Records are also not deleted when consumed. The documentation is explicit that events “are not deleted after consumption” and that retention is a per-topic setting. That single property is what makes replay possible and what makes a topic usable by several independent consumers.

Consumer groups: the trade between order and throughput

Consumers coordinate through groups, and the design documentation states the rule that follows from the log structure: the topic is “divided into a set of totally ordered partitions, each of which is consumed by exactly one consumer within each subscribing consumer group at any given time.”

• Parallelism is capped by partitions. Ten partitions support at most ten working consumers in a group; the eleventh sits idle. Scaling consumption means repartitioning, which is not free.
• Order is preserved per partition, and a key inherits it by being routed to one. More partitions buy throughput without weakening any single key’s ordering — a key still occupies one partition. What does break it is changing the routing: adding partitions, switching the partitioner, or specifying a partition explicitly sends that key’s new records somewhere else, leaving its history split across two ordered sequences. Plan that migration rather than discovering it.
• Position is just a number. Because a consumer’s position in a partition is a single integer, the broker keeps very little state — and the documentation notes the useful consequence: “a consumer can deliberately rewind back to an old offset and re-consume data.”

That last point is the difference from a queue that deletes on acknowledgement. Reprocessing after a bug fix is a normal operation rather than a recovery project — provided the destination tolerates seeing the same records again, which is the delivery semantics question rather than a Kafka setting. How far behind each group is running is the number to monitor: see committed offset and consumer lag.

Replication: what “written” means

Each partition has one leader and zero or more followers; all writes go to the leader, and followers pull from it like ordinary consumers. The set of replicas that are keeping up is the in-sync replica set (ISR). Brokers stay in it by maintaining an active session with the controller and by not falling too far behind — a replica that cannot catch up within replica.lag.time.max.ms is removed.

Two documented rules do most of the work in practice. “Only committed messages are ever given out to the consumer,” so consumers never see a record that could vanish in a leader failure. And the durability promise is conditional: “a committed message will not be lost, as long as there is at least one in sync replica alive, at all times.”

Producer acks	What the producer waits for	What it risks
0	Nothing	Records can be lost without the producer knowing
1	The leader only	A leader failure before replication loses the record
all	The full in-sync set	Higher latency; writes fail when the ISR is too small

The setting that makes acks=all meaningful is the topic’s min.insync.replicas. With a replication factor of three and a minimum of two, a write succeeds while one replica is down and fails when two are — which is the intended behavior, and a surprise to teams who expected the cluster to keep accepting writes.

And when every replica of a partition dies, there is no good option, only a choice: wait for an in-sync replica to return, or promote whatever comes back first and accept that its log may be missing committed records. Kafka’s documentation describes this as “a simple tradeoff between availability and consistency,” and states that from version 0.11.0.0 it defaults to waiting for a consistent replica, changeable through unclean.leader.election.enable. What that switch actually costs is worked through in unclean leader election.

Retention: two ways to forget

	Time or size based deletion	Log compaction
Keeps	Everything within the window — log.retention.hours defaults to 168, that is seven days	At least the last known value for each message key
Suits	Event streams where each record stands alone	Change streams for keyed, mutable data
Lets a consumer	Replay a recent window	Rebuild full current state from the topic alone

Retention is where a streaming platform quietly becomes a source of truth or fails to be one. A seven-day window means a consumer that has been down for eight days cannot resume — it must be re-seeded from somewhere else. A compacted topic, by contrast, is documented as letting “downstream consumers restore their own state off this topic,” which is what makes it usable for caches and materialized state.

Rebuilding state from a compacted topic comes with one condition that is easy to miss, and getting it wrong leaves deleted keys alive. A delete is represented by a record with a null value, and those markers do not live forever: the documentation notes that “delete markers are special in that they will themselves be cleaned out of the log after a period of time to free up space,” governed by delete.retention.ms. The guarantee is conditional on speed — “all delete markers for deleted records will be seen, provided the consumer reaches the head of the log in a time period less than the topic’s delete.retention.ms setting.”

So the two recovery situations are not the same. Rebuilding from empty works if the read finishes inside that window; a full restore that takes longer than the marker retention can apply a key’s last value and never see the delete that followed it. Resuming a consumer that has been stopped for longer is worse, because it keeps state it built earlier and the markers that would have removed keys are already gone — those keys stay in the state with no error anywhere. The practical rules: set delete.retention.ms against how long a full restore actually takes, and treat a consumer stopped longer than that as needing a rebuild from empty rather than a resume. Note also that cleanup.policy=compact keeps the latest value for every key indefinitely, while compact,delete also expires old segments by time — under which a rebuild can miss keys whose only record aged out.

Neither policy is an archive. Compaction guarantees the latest value per key, not the history of how it got there; deletion guarantees a window, not a record of everything that happened in it.

The event contract is the hard part

Nothing above concerns what is in the records, and that is where most long-lived pain accumulates. A topic that several teams consume is an interface, and it evolves under the same rules as any other: readers and writers upgrade at different times, so changes must be compatible in a direction someone has decided on. That is the subject of schema evolution and compatibility, usually enforced through a schema registry and agreed as a data contract.

Three decisions are worth making explicitly when a topic is created rather than when it breaks: what the key means, because it determines ordering and compaction; what a record represents — a state change, a completed fact, or a command; and who may add fields, remove them, or change their meaning. See designing topics and event contracts.

What Kafka does not solve

• Writing to a database and publishing an event are still two operations. A service that commits a row and then sends a message can crash in between, leaving the two out of step. The standard remedy is to write the event in the same transaction as the data and publish it afterwards — the transactional outbox.
• Transactions inside Kafka do not extend to your database. Atomic writes across partitions are a Kafka-side facility; coordinating them with an external system’s commit is a separate design problem.
• Cross-partition order does not exist, so a business process whose steps land in different partitions has no global sequence to rely on. Either key the events so related ones share a partition, or make the consumer tolerate arrival order — the same problem last writer wins describes.
• Duplicates are still the destination’s problem. Redelivery is normal, so the consumer’s writes need to be idempotent or deduplicated — see event deduplication and the idempotent producer for the two ends of that.
• Availability under network partitions is bounded. The documentation says it directly: Kafka “may not remain available in the presence of network partitions” — the familiar trade that quorum-based systems all make.

None of these are defects. They are the line where an infrastructure guarantee ends and a design decision begins, and the projects that go badly are usually the ones that assumed the line was somewhere else.

References

Apache Kafka 4.3.X documentation was checked in September 2026. Defaults such as retention and leader-election behavior change between versions; confirm them for the version you run.

• Apache Kafka documentation, Introduction
• Apache Kafka documentation, Design (replication, consumer position, log compaction)
• Apache Kafka documentation, Broker Configs