Kafka Fundamentals

In plain terms

An order service publishes an event once; a fraud service and a warehouse loader each read it at their own pace. Kafka retains records according to topic policies, independently of these readers, so they can resume or replay the history that remains available. Its central structure is a distributed log: records are appended in order within each partition, with durability determined by replication and acknowledgement settings.

Earlier articles introduced streaming collection and log collectors. Kafka can be the retained handoff between those collectors and downstream processing. Here we follow partition placement, acknowledgements, consumer progress, and replay, including their failure conditions.

A log, not a queue

In a conventional work queue, acknowledgement usually removes a message from that queue. Fan-out queues and messaging services with retention can also support multiple readers or replay. Kafka makes the retained log explicit: each partition assigns offsets to appended records, and consumer groups track progress independently. Reading does not delete a record. Replay starts at an available offset, not necessarily offset zero; deletion and compaction policies determine which history remains.

Three properties follow, and they are the reasons a platform puts a log between its sources and its consumers. Decoupling: the checkout service writes once and does not know or care that the fraud model, the warehouse loader, and the analytics job all read it. Buffering: when a consumer is down or slow, records wait in the log, not in the producer’s memory, provided retention and storage capacity cover the outage; a broker outage still requires producer-side retry or buffering. Replay: a new consumer, a fixed consumer, or a backfill reads history from the log through the same code that reads the present, which is what the Kappa idea in the architecture article depended on.

Topics, partitions, offsets, keys

A topic is a named log, such as events.checkout or cdc.orders. A topic is split into partitions, each an independent append-only log with its own offsets, spread across the brokers, the servers that store them. Partitions are the unit of everything: of parallelism, because different consumers read different partitions; of ordering, because records are ordered within a partition and nowhere else; and of placement, because a record goes to one partition and stays there.

With stable key serialization, partitioner, and partition count, key-based placement sends the same key to the same partition. Kafka preserves that partition’s append order, not global event-time order. Concurrent producers can interleave, and retries without suitable ordering settings can reorder sends. Different keys may share a partition; they then share its log order. Null-key placement depends on the client and partitioner and often batches records using sticky placement rather than strict round-robin. Choose a key for the entity whose updates must stay together. The general risk is splitting one key’s old and new history across partitions.

A record is a key, a value, a timestamp, and optional headers. Identify its log position by topic, partition, and offset; offset 10 in partition 0 is unrelated to offset 10 in partition 1. The broker stores the value as bytes rather than enforcing the application’s business schema; that schema is the producer’s and consumer’s agreement.

Replication and acknowledgement

A partition has a leader replica and follower replicas on other brokers. Writes go to the leader; consumer reads normally do too, although configured follower reads are possible. The in-sync replica set (ISR) includes the leader and followers meeting Kafka’s catch-up criteria; it does not mean every follower has every new record at every instant. On leader failure, a suitable replica can take over. Safe election policy and surviving replicas matter to whether acknowledged data remains available.

acks=0 waits for no broker acknowledgement, so the producer cannot confirm acceptance. acks=1 confirms the leader’s append; failure before replication can lose that record. acks=all waits for the ISR, while min.insync.replicas=2 rejects such writes when fewer than two replicas are in sync. With three replicas, this can tolerate one broker failure if the remaining replicas and controller quorum are available. Leader election and retries may still interrupt requests; it is not a zero-downtime or fixed-latency promise. An acknowledgement confirms the configured replication condition, not an fsync on every disk or completion in an external database. A timeout is ambiguous: the append may have succeeded.

Producers

A producer batches records per partition and can compress batches before sending them. linger.ms allows time to fill batches; throughput also depends on record size, network, broker capacity, and workload. Lost acknowledgements make retries ambiguous. Kafka’s idempotent producer uses producer identity, epochs, and per-partition sequences to recognize retries within its protocol. For the Java 4.1 client, idempotence is enabled by default when settings do not conflict; it requires acks=all, retries, and at most five in-flight requests per connection. Check the actual client and configuration, and handle terminal send failures. This is not deduplication of a business event sent again as a new record or after an application restart.

Consumers, groups, and offsets

A conventional consumer group divides assigned partitions among members: a partition has at most one assigned member at a time, and members may own several partitions. More members than assigned partitions leaves some idle. This assignment does not guarantee one processing attempt: crashes and rebalances can cause redelivery. A separate group has separate progress and can read the same retained records. This article uses conventional consumer groups, not Kafka share groups with per-record acknowledgement.

Distinguish position, the next offset this consumer will fetch, from the committed offset, the restart position stored for the group and partition. Polling advances position; committing saves progress and does not itself seek the reader. After processing offset 49 durably, commit 50: the next offset to read. Reassignment resumes from committed progress. Commit a whole batch first and a crash can skip unfinished work; commit after durable output and a crash before commit can repeat work. Under these assumptions, these are at-most-once and at-least-once processing strategies, respectively. Automatic commit is not inherently commit-before-processing: safety depends on the client’s poll/commit behavior and whether previously fetched work has completed. Asynchronous handoff can commit ahead of durable work. Use explicit commits when coordinating output, and commit only through the contiguous completed prefix of each partition.

Suppose one partition returns offsets 50, 51, and 52. If 50 and 52 finish but 51 is still running, commit 51, not 53: the committed value names the next record to resume. Committing 53 would skip unfinished 51 after a restart. Once 51 finishes durably, the completed prefix extends through 52 and can advance to 53. Keep completion tracking per partition, and stop stale workers from committing or producing uncontrolled effects after reassignment.

Committed-offset lag is the distance from a partition’s end to the group’s committed offset. Inspect it per partition, together with processing latency, oldest unfinished work, and error rates. Offset distance need not equal a count of readable records after compaction or transactions. One hot partition can dominate completion time. Speeding up that partition’s consumer or output can help; adding group members alone cannot divide its assignment. Splitting a hot key trades away the original per-key order unless another mechanism restores it.

Retention, compaction, replay

Delete retention removes eligible old log segments according to time and per-partition size limits, independently of consumer progress. Verify topic overrides and broker defaults; deletion is not an exact per-record TTL. If a saved offset is no longer available, auto.offset.reset determines whether to start at the earliest available offset, at the end, or raise an error, depending on the selected policy. It also applies when no committed offset exists; it does not rewind a valid committed offset on every restart. Resetting to the earliest preserves only the remaining history. A gap requires recovery from another source or explicit acceptance of missing data.

Compaction asynchronously removes superseded values for a key. It preserves record offsets and order, leaving gaps; it does not instantly turn a topic into a one-row-per-key table. A null-valued record, a tombstone, represents deletion, and delete markers can later expire. A state-building consumer must account for that retention window. With compaction alone, the latest non-deleted state can be retained; combining compact,delete also permits time/size deletion. Replaying a compacted topic reconstructs retained state, not every historical event. Tiered storage changes where segments live, not whether retention can remove them; replay also needs compatible historical schemas and controlled output effects.

If a consumer retains an old local row but misses the tombstone before it expires, resuming the surviving compacted log may leave that stale row in place. Restoring a current-state view may require rebuilding an empty target from a complete retained state or reconciling against an authoritative snapshot. Replaying a suffix into an old target is not equivalent to rebuilding from scratch.

Delivery, end to end

Transactions can atomically publish output records and consumed offsets within Kafka. Downstream consumers need isolation.level=read_committed to exclude aborted transaction records. The transaction boundary is the key: a separate database transaction followed by a Kafka offset commit still leaves a crash window. For an external sink, use a durable event ID with atomic deduplication and the business update, or a supported coordination protocol. Producer idempotence suppresses protocol retries, while consumer retries and business duplicates require their own handling. Eventual delivery also assumes successful recovery before the required data expires.

Kafka, Redpanda, Kinesis, Pub/Sub

These systems all decouple event producers and consumers, but they do not expose identical delivery or replay models. The table is a starting comparison, not a feature-compatibility guarantee. Confirm client support, quotas, retention configuration, and regional availability for a deployment.

	Apache Kafka	Redpanda	Amazon Kinesis Data Streams	Google Pub/Sub
What it is	the original; self-managed, or managed (Confluent, MSK, and others)	a Kafka-protocol-compatible broker in C++, one binary, Raft, no JVM	AWS’s managed stream service with its own API	Google’s managed messaging service with its own API
Unit of ordering and parallelism	partition	partition	shard	no exposed partitions; per-key order when enabled and published in one region
Position	offset per partition, committed by the group	same	sequence number per shard, checkpointed by the consumer application	acknowledgement per message per subscription; no offsets
Retention	configurable; tiered storage still follows retention policies	configurable, tiered storage built in	24 hours by default, extendable to a year	up to 31 days configured retention; time seek needs retained messages; snapshots have separate expiry
Scaling	add partitions and brokers; the partition-count trap applies	same	split and merge shards; throughput fixed per shard (on-demand mode scales shards for you)	automatic service scaling; quotas and subscriber capacity still matter
Operations	brokers, disks, rebalances, upgrades, capacity; or a managed service’s bill	deployment-specific operations; verify compatibility and recovery	none of the broker operations; service quotas instead	subscriber health, quotas, IAM, retention, and cost
Ecosystem	Connect, Streams, Flink, Debezium, Schema Registry, many client libraries	the Kafka ecosystem, through the protocol	AWS integrations; Kafka tools only through bridges	Google integrations; Kafka tools through bridges
Fits when	replay, long retention, and the ecosystem matter; a team can run it or pay for it	the Kafka ecosystem is wanted with a smaller operational footprint	AWS integration and its stream API match the workload	managed messaging and subscription semantics match the workload

Choose by required retention, replay procedure, client compatibility, and operational ownership. A managed service runs the brokers, but the application team still owns schemas, keys, consumers, permissions, and recovery. Pub/Sub acknowledgement and seek cannot be treated as renamed Kafka group offsets.

Operating a cluster

Monitor under-replicated and offline partitions, per-group lag, request errors and latency, and storage headroom. Retained bytes roughly follow ingress rate times retention times replication factor, with compression, indexes, segment behavior, and spare capacity accounted for; consumer fan-out adds read and network load. Kafka 4.x uses a KRaft controller quorum for cluster metadata and leadership coordination, distinct from data-partition replicas. The controller quorum needs an available majority; plan its failures separately from data replication. TLS encrypts traffic and can authenticate clients with certificates; SASL provides configured authentication, and ACLs authorize resource operations. Apply producer/consumer quotas and record topic ownership and configuration.

The subscription company’s topics

The company here is a hypothetical subscription business used as an illustrative case. The topic names, partition counts, and retention periods below are proposed example settings, not a description of a real organization’s cluster or a recommendation for every workload.

Topic	Key	Partitions	Retention	Producers, consumers
events.checkout	anonymous or user id	24	30 days total; optionally tier older retained segments to object storage	the collector aggregator; the raw-layer writer, the fraud lane, the funnel job
cdc.orders	order id	12	14 days	the CDC connector from the database article; the warehouse merge consumer
cdc.customers	customer id	6	compacted	the CDC connector; any service that needs the current customer row
logs.app	none	48	3 days	every collector; the log index

Keys follow what must be ordered, partition counts follow expected throughput with room to grow, retention follows who needs to replay and how far back, and the compacted topic is the one that carries state.

Anti-patterns

• Ordering assumed across partitions. A consumer that expects the checkout to arrive after the product view for a different key.
• Growing partitions casually. Doubling the count on a keyed topic and splitting some customers’ histories across partitions.
• acks=1 for events that matter. A leader failover during a busy hour and a few thousand orders that were acknowledged by the leader but lost before replication.
• Committing ahead of durable work. A fetched batch handed to asynchronous workers and committed before their outputs finish.
• More consumers than partitions. Scaling the group to twenty on a six-partition topic and wondering why lag did not move.
• One hot key. The biggest customer, or a null key replaced by a constant, in one partition that decides every job’s finish time.
• Retention shorter than the longest outage. A consumer down for a weekend on a topic retained for two days.
• The queue mindset. Deleting a topic’s records after reading them, and losing the replay that was the point.

References

Kafka behavior and Java client defaults above are scoped to the linked 4.1 documentation. Service documentation checked on 2026-09-09; verify the version and service settings used in your deployment.

• Kafka 4.1 design: replication, transactions, compaction
• Kafka 4.1 producer configuration
• Kafka 4.1 consumer configuration
• KafkaConsumer API: position, commits, rebalances
• Kafka 4.1 KRaft operations
• Kinesis Data Streams concepts and retention
• Pub/Sub topic retention
• Pub/Sub replay and seek
• Pub/Sub message ordering
• Redpanda introduction