How Many Partitions

A cold-chain logistics company runs 6,000 refrigerated trucks. Every 2 seconds, each truck sends a telemetry reading of about 1.5 KB with the cargo temperature, the door state, the location, and engine data. That is 3,000 readings a second, or about 4.5 MB a second, written to a topic named truck-telemetry.

Three services read the topic: an alert checker that warns when cargo warms up, a live map, and an archive. The company expects the fleet to triple within two years. The partition count must be chosen when the topic is created. Kafka lets the team add partitions later, but adding them has side effects, and removing them is not possible.

This page turns those figures into a partition count, states every assumption behind it, and shows what a later change of count does to the data.

What the Count Limits

The partition count is the number of logs a topic is split into, and it caps how many consumers in one group can read the topic at the same time. Kafka's design documentation states that each partition is consumed by "exactly one consumer within each subscribing consumer group at any given time." On a topic with 6 partitions, a group can therefore keep at most 6 consumers busy; a seventh would receive nothing. The operations guide says it directly: "the partition count impacts the maximum parallelism of your consumers."

The count also limits how far the topic spreads across brokers. The same guide notes that "each partition must fit entirely on a single server," so a topic's write load can be spread over at most as many brokers as it has partitions. The partitions themselves are the topic's logs, as described in Topics and the Log.

The count therefore has to satisfy two needs: enough partitions to accept the writes, and enough partitions for the slowest reader to run enough consumers.

Sizing by Data Rate

Throughput per partition and per consumer depends on the hardware, the record size, and what the consumer does with each record, so the figures below are assumptions for the arithmetic. Measure your own before you rely on any of them.

• Writes. Suppose one partition accepts about 10 MB a second of writes on these brokers.
• The slowest reader. The alert checker writes every reading to a database, and suppose one instance handles about 500 readings a second. At 1.5 KB a reading, that is 0.75 MB a second.
• The other readers. Suppose one live map instance handles about 3 MB a second, and the archive writes large batches and keeps up with one instance.

The count is the larger of two divisions: the write rate over what one partition accepts, and the write rate over what one consumer of the slowest group handles. Every group reads the whole topic, so each group must keep up with the full 4.5 MB a second.

The reads decide this topic, not the writes. At 10 MB a second per partition, the writes need only 2 partitions even after the fleet triples, while the alert checker needs 18 consumers to keep up.

The last row adds room to catch up. A consumer group running at full capacity can keep up with new readings, yet it can never shrink a backlog. If the alert checker is down for 10 minutes at 13.5 MB a second, about 8.1 GB waits for it. With 24 consumers each running at 75 percent of capacity, the group has 4.5 MB a second spare, so it clears that backlog in about 30 minutes. With exactly 18, it never would.

The Cost of Too Many

More partitions are not free. Each partition is a log in its own folder on a broker, and with the documented durable setup of three copies, 24 partitions are 72 logs to store, replicate, and move when a broker fails.

The producer is affected too. It groups records into batches, and a request to a broker carries "one batch for each partition with data available to be sent." If the team instead created 48 partitions, twice the 24 it needs, the same 4.5 MB a second would give each partition's batch about half the records, so the producer would send smaller batches for the same data.

Readers rarely benefit beyond their need. The live map needs 2 consumers today and the archive needs 1, so most of the 24 partitions give them nothing extra. Size the topic for its slowest reader at its planned load, not for a guess at many times that load.

Adding Partitions Later

A topic that turns out too small can be grown. Kafka's operations guide describes the cost, because the default partitioner places each key by "hash(key) % number_of_partitions." When the count changes, the result of that division changes too, so "messages with the same key may be routed to different partitions after the expansion," and Kafka "will not attempt to automatically redistribute existing data."

The program below shows both effects on a real broker. It writes one reading for each of 1,000 trucks to a topic with 6 partitions, grows the topic to 18, and writes the same 1,000 trucks again. Finally it asks Kafka to go back to 6.

// Growing a topic: allowed. Existing records stay where they are.
admin.createPartitions(Map.of(TOPIC, NewPartitions.increaseTo(18))).all().get();
// ...
long moved = before.keySet().stream().filter(k -> !before.get(k).equals(after.get(k))).count();
// ...
// Shrinking a topic: refused.
admin.createPartitions(Map.of(TOPIC, NewPartitions.increaseTo(6))).all().get();
Copy

Running it against a single Kafka 4.4 broker printed the following.

Output:

first round: 1000 trucks written to 6 partitions
partitions grown from 6 to 18
records still in partitions 0 to 5: 1000
second round: 649 of 1000 trucks now write to a different partition
reduce to 6: InvalidPartitionsException: The topic truck-telemetry currently has 18 partition(s); 6 would not be an increase.
Copy