Unclean Leader Election

Unclean leader election is promoting a replica that is not known to be up to date into the leader role, because no up-to-date replica is available. The name marks the compromise: the new leader’s log becomes the truth even though it may be missing records that were already acknowledged as committed. Descriptions follow Apache Kafka 4.3.X documentation, where the behavior is controlled by unclean.leader.election.enable.

The situation it arises in

Kafka’s durability promise is conditional: a committed message will not be lost as long as at least one in-sync replica stays alive. The documentation states the boundary plainly — if all the nodes replicating a partition die, “this guarantee no longer holds.” The same fork arises whenever no safe candidate survives — in Kafka 4.x that means the in-sync replicas and the eligible leader replicas (ELR), replicas outside the ISR that are still known to hold every committed record — and only lagging replicas remain. At that point only two behaviors are possible:

The documentation calls this “a simple tradeoff between availability and consistency” and notes that it is not specific to Kafka: any quorum-based scheme faces the same fork when a majority is permanently lost — lose data, or keep serving from an incomplete survivor.

What the default says

Per the documentation, from version 0.11.0.0 Kafka chooses the first strategy and favors waiting for a consistent replica, with the setting available “to support use cases where uptime is preferable to consistency.” In other words the product’s default is: stay down rather than serve a log with holes, and the switch exists for people who have decided otherwise.

Framing it as a configuration understates it. Enabling unclean election is a written decision that, under a specific failure, the organization prefers losing acknowledged records to being unavailable. That is a reasonable choice for a topic of metrics samples and a serious one for a topic of payments — and the difference means it belongs to the topic’s owner rather than to a cluster-wide default someone set once.

Before touching it

• Check the cheaper defenses first. A higher replication factor and replicas spread across failure domains make it less likely that every replica of a partition is lost together. min.insync.replicas works at a different point: with acks=all it sets how many in-sync replicas must hold a write before it is acknowledged, which limits how few copies an acknowledged record can have, at the price of rejecting writes when too few replicas are in sync.
• Know what silent loss looks like downstream. Nothing in the protocol tells an application which business records were lost. A consumer whose committed offset no longer exists in the truncated log applies auto.offset.reset: earliest or latest moves it to a new position without an error, while none throws an exception that reports a missing offset, not which records are gone. Offsets that belonged to lost records can also be reused by new records, so the loss does not necessarily appear as a gap in offsets. Reconciliation against the source is what confirms what was lost.
• Decide per topic, and record why. A value chosen during an incident, on the argument that the cluster must come back now, is the version of this decision nobody wants to defend afterward.

How this fits with in-sync replicas, acknowledgments, and the rest of Kafka’s replication model is worked through in Kafka as a Log; for the underlying idea, see quorum.

References:

• Apache Kafka 4.3 documentation, Eligible Leader Replicas
• Apache Kafka documentation, Design (Replication)
• Apache Kafka 4.1 Documentation, Design