Idempotent Kafka Producer

An idempotent Kafka producer uses protocol identity, epochs, and per-partition sequence numbers to recognize supported retries. If the broker appended a record but the acknowledgement was lost, retrying through that protocol can avoid appending another copy.

For the Java 4.1 client, compatible settings require acks=all, retries, and no more than five in-flight requests per connection. Defaults depend on the client and version, and terminal failures still require handling. Do not apply a tiny teaching model’s one-pending-record rule to Kafka’s actual implementation.

Sending the same business event as a new application call can create another record even when protocol retries are idempotent. Consumer replay is another source of repeated work. Use durable business-event identity and coordinated destination writes when those effects must be deduplicated; producer idempotence does not make an external payment or email occur exactly once.

Reference: Kafka fundamentals lab.