Partition Key

A partition key is the record field or derived value used by a key-based partitioner to choose a partition. In Kafka, a producer commonly hashes the serialized key bytes. An explicit partition or a custom partitioner can change that behavior, so record the actual routing policy rather than infer it from a field name.

Using an order ID can keep that order’s records together while distributing different orders. A customer ID groups all of a customer’s orders, but one unusually active customer can dominate a partition. Splitting that customer with an order-derived salt can improve distribution while giving up customer-wide ordering. Test the busiest partition, not only the average load.

Colocation supports reading partition append order; it does not reconstruct when events happened across concurrent producers. Keep serialization and partitioning consistent across writers. Changing the partition count can remap a key, and changing its encoded bytes can affect both routing and compaction identity. A tombstone written under a different byte key does not delete the original key.

For a migration, define the boundary between old and new routing, how consumers reconcile overlapping histories, and how old keyed state is removed. The illustrative CRC32 hash in the lab is not Kafka’s Java default partitioner.

Try the examples in Designing Topics and Event Contracts.

Reference: Kafka design.