Choosing a Partition Key

A marketplace publishes every event of an order, from OrderPlaced through OrderPaid and OrderPacked to OrderShipped, to a topic named marketplace-orders, which has 8 partitions. The events are keyed by seller ID. After 10,000 orders, a count of OrderPlaced events per partition printed this:

key = seller ID
  partition 0:   825 orders
  partition 1:   750 orders
  partition 2:   850 orders
  partition 3:   850 orders
  partition 4:  4225 orders
  partition 5:   700 orders
  partition 6:   775 orders
  partition 7:  1025 orders
  busiest partition: 42.3% of orders (an even spread is 12.5%)
Copy

Partition 4 holds more than five times as many orders as partition 5. The cause is a single seller, S-0007, which sells 35 percent of everything on the marketplace. Every one of its orders carries the same key, so every one of them goes to the same partition, and the consumer that reads partition 4 does more work than any other consumer in its group.

This page explains how to choose a key, how to estimate a large key's share of the load before it causes trouble, and what each repair costs.

What a Key Decides

The key was introduced in Designing an Event as the value that keeps related events in order. The producer hashes it to choose a partition, so a key decides two things at once: which events stay in order together, and how the load spreads across partitions.

A hot key is a key that carries a much larger share of a topic's events than the average key, so the partition it hashes to receives more than its share of the load. Kafka does not balance this. The default partitioner chooses "a partition based on a hash of the key," and each partition is read by exactly one consumer in a group, so one busy key means one busy consumer, however many other consumers the group has.

Three services read marketplace-orders:

• Fulfillment handles each order's four events from this topic in sequence: placed, paid, packed, shipped.
• Seller dashboards show each seller's sales for the day.
• Fraud scoring checks each order on its own.

Only fulfillment needs order, and it needs it per order, not per seller. Two orders from S-0007 can be packed in either sequence without harm. The seller ID was chosen because it seemed natural, not because any reader needed per-seller sequence.

Estimating a Hot Key's Share

The busiest partition's share can be estimated before any code runs. If the largest key carries a share s of the events, and the other keys spread evenly over N partitions, the partition holding the large key receives about s + (1 - s) / N of all events.

For the marketplace, s is 0.35 and N is 8, so the busiest partition receives about 0.35 + 0.65 / 8, or 43 percent. The run measured 42.3 percent, because the small sellers did not spread perfectly evenly.

The share becomes a problem when it is compared with what one consumer can handle. Suppose the marketplace peaks at 2,000 orders a second, and one fulfillment consumer handles about 400 orders a second. At peak, partition 4 receives about 845 orders a second, more than twice that capacity. Its backlog grows by about 445 orders a second, or about 1.6 million an hour, while each of the other seven consumers runs at about half its capacity or less.

Adding partitions does not fix a hot key. With 32 partitions the estimate is 0.35 + 0.65 / 32, still about 37 percent, because S-0007 alone is 35 percent of the load and all of it lands on one partition. The fix has to change the key.

The input to this arithmetic is the share of traffic held by the largest keys. It can usually be read from existing data, for example by counting last month's orders per seller in the orders database, before the topic is created.

Three Keys, One Run

The program below sends the same 10,000 orders three times, each time with a different key, and counts how many land in each partition. Seven of every 20 orders belong to S-0007, and the rest come from 260 small sellers. The run sends one OrderPlaced per order; each order's later events carry the same key, so they follow it to the same partition and the counts show how the whole load spreads.

run("key = seller ID", i -> sellerOf(i));
run("key = order ID", i -> "O-" + (500000 + i));
// Only the known large seller is split, into 4 sub-keys; small sellers keep their plain key.
run("key = seller ID, large seller split 4 ways", i -> sellerOf(i).equals(BIG_SELLER) ? BIG_SELLER + "#" + (i % 4) : sellerOf(i));
Copy

Running it against a single Kafka 4.4 broker printed the following.

Output:

key = seller ID
  partition 0:   825 orders
  partition 1:   750 orders
  partition 2:   850 orders
  partition 3:   850 orders
  partition 4:  4225 orders
  partition 5:   700 orders
  partition 6:   775 orders
  partition 7:  1025 orders
  busiest partition: 42.3% of orders (an even spread is 12.5%)
key = order ID
  partition 0:  1311 orders
  partition 1:  1281 orders
  partition 2:  1243 orders
  partition 3:  1260 orders
  partition 4:  1242 orders
  partition 5:  1202 orders
  partition 6:  1241 orders
  partition 7:  1220 orders
  busiest partition: 13.1% of orders (an even spread is 12.5%)
key = seller ID, large seller split 4 ways
  partition 0:   825 orders
  partition 1:   750 orders
  partition 2:  1850 orders
  partition 3:  1850 orders
  partition 4:  1725 orders
  partition 5:  1200 orders
  partition 6:   775 orders
  partition 7:  1025 orders
  busiest partition: 18.5% of orders (an even spread is 12.5%)
Copy

The order ID spreads the load almost evenly: the busiest partition holds 13.1 percent, close to the ideal 12.5. Splitting S-0007 into four sub-keys spreads its orders over partitions 2, 3, 4, and 5, and the busiest partition falls to 18.5 percent. At 2,000 orders a second, that is about 370 a second. That is under one consumer's 400, but it is about 92 percent of that capacity, which leaves little room to catch up after an outage. A real design would split S-0007 into more sub-keys and check the result the same way. The four sub-keys are also uneven: because the bucket comes from the order number, they received 1,000, 1,000, 1,000, and 500 of S-0007's 3,500 orders.