Topics and the Log

A Kafka topic, the named stream of events introduced in Requests and Events, is not stored as one long sequence. Inside the brokers, a topic is split into a fixed number of partitions. Each partition is a log of its own: an ordered sequence of events in which new events are only ever added at the end, and each event receives the next offset in that partition.

Consider a food delivery platform. For every order it publishes OrderPlaced, CourierAssigned, OrderPickedUp, and OrderDelivered. Separately, the phone of every courier on shift reports its location every 4 seconds. Someone has to decide how many topics these events become, and which of them share a topic.

The sections below explain what a partition stores and promises, and then use that to decide which events belong together in one topic.

Partitions

A partition is an ordered, append-only log that holds part of a topic's events, and every event in a topic is stored in exactly one partition. Kafka's documentation states that events "are organized and durably stored in topics," and that when an event is published, "it is actually appended to one of the topic's partitions." A topic's partitions can sit on different brokers, which is how one topic is spread over several machines.

The producer, not the broker, chooses the partition. When an event has a key, the default partitioner chooses "a partition based on a hash of the key," so every event with the same key goes to the same partition. That is why Designing an Event keys events by the thing whose events must stay in order. The platform keys every order event by its order ID, so all four events of one order are stored together.

Three properties follow from this structure:

• Offsets belong to a partition. Requests and Events described an offset as an event's position in the topic; more precisely, it is a position within one partition. Each partition numbers its own events from 0, so a topic with three partitions has three different events at offset 0. A consumer's position is likewise "just a single integer" per partition.
• Order exists only inside one partition. Kafka guarantees that a consumer of a partition reads "that partition's events in exactly the same order as they were written." Nothing is promised about two events in different partitions.
• A partition is a unit of storage. Each partition is kept in its own folder under the broker's log directory, so the partitions of one topic can be stored and served by different machines.

Reading the Log

The program below writes the four lifecycle events of four orders to delivery-events, a topic with three partitions. The orders move forward together, so their events are written interleaved: all four OrderPlaced events first, then all four CourierAssigned events, and so on.

for (String type : steps) {
    for (String order : orders) {
        producer.send(new ProducerRecord<>(TOPIC, order, type)).get();
    }
}
Copy

It then reads each partition from its first offset. A courier payouts service needs only OrderDelivered, so the program also counts how many events that reader had to read to find them.

// A payout service needs only deliveries, but it must read every event to find them.
if (r.value().equals("OrderDelivered")) used++;
Copy

Running it against a single Kafka 4.4 broker printed the following.

Output:

partition 0 holds 8 events
  offset 0: D-6624 OrderPlaced
  offset 1: D-6627 OrderPlaced
  offset 2: D-6624 CourierAssigned
  offset 3: D-6627 CourierAssigned
  offset 4: D-6624 OrderPickedUp
  offset 5: D-6627 OrderPickedUp
  offset 6: D-6624 OrderDelivered
  offset 7: D-6627 OrderDelivered
partition 1 holds 4 events
  offset 0: D-6625 OrderPlaced
  offset 1: D-6625 CourierAssigned
  offset 2: D-6625 OrderPickedUp
  offset 3: D-6625 OrderDelivered
partition 2 holds 4 events
  offset 0: D-6626 OrderPlaced
  offset 1: D-6626 CourierAssigned
  offset 2: D-6626 OrderPickedUp
  offset 3: D-6626 OrderDelivered
courier payouts read 16 events and used 4
Copy

Four details in this output matter for design:

• Each partition counts from 0. Partitions 1 and 2 hold offsets 0 to 3, while partition 0 holds offsets 0 to 7.
• One partition can hold several keys. Orders D-6624 and D-6627 hashed to partition 0, so their events alternate there. Each order's own four events still appear in the order they were written.
• Partitions are not always even. Partition 0 holds twice as many events as the others, only because two keys hashed to it.
• A reader receives every event in the topic. The payouts reader used 4 of the 16 events and skipped the other 12.

What a Topic Decides

Several settings belong to the topic rather than to individual events, so every event in a topic shares them:

• Retention. Kafka keeps events for a period set "through a per-topic configuration setting," 7 days by default, and then discards them.
• Cleanup policy. A partition's log is stored on disk as a series of files called segments. The topic either deletes old segments, which is the default, or compacts them. Compaction keeps at least the latest event for each key.
• Partition count. Every event type in the topic is spread over the same partitions.
• Readers. A consumer subscribes to a whole topic, so it receives every event type stored there.

Putting two kinds of events in one topic therefore forces them to share all four. Splitting them into two topics gives each its own settings, at the price of the order between them.

One Topic or Many

Events that must be read in order relative to each other must share a topic and a key. Order exists only inside a partition, and a partition belongs to one topic. Suppose OrderPlaced lived in a topic named order-placed and OrderDelivered in a topic named order-delivered. A support dashboard that reads both could process the delivery of D-6625 before its placement, because Kafka reads the two topics independently and orders nothing between them.

The four lifecycle events pass this test. They describe the same order, they share the order ID as their key, and their readers, such as the support dashboard and the customer app, need them in sequence. They belong together in delivery-events.

Courier locations are a different stream, and the numbers show it. Suppose 9,000 couriers are on shift, and each sends a location of about 200 bytes every 4 seconds. That is 2,250 events a second, about 450 KB a second, or about 39 GB a day. Now suppose the platform handles 300,000 orders a day, with four lifecycle events of about 500 bytes each. That is 1.2 million events, or about 600 MB a day, which averages about 14 events a second.

The two streams also need different retention. Order events are kept 30 days so that support can settle disputes, which with three copies of every event is 600 MB x 30 x 3, about 54 GB. Locations feed only the live map, so 1 day is enough, which is 39 GB x 1 x 3, about 117 GB.

In one shared topic, both streams would need the longer retention, and 30 days of locations is 39 GB x 30 x 3, about 3.5 TB. The payouts service would also read about 650 events for every delivery it needs, because locations outnumber deliveries by that much. Locations are about a courier rather than an order, so they would also need a different key. These are the signs that two streams belong in separate topics: a different rate, a different retention, a different key, or different readers.

Comparing the Two Layouts

The first layout costs readers extra work. In the run, payouts skipped three events for every one it used, and in a busy topic that ratio is paid in network and processing time. The second layout costs order, which matters most when a reader builds state from a sequence, such as an order status page.

When event types have no order relation, separate topics are better. A MenuUpdated event from a restaurant and an OrderDelivered event share no key and no readers, so nothing is gained by storing them together.

Key Takeaways

• A topic is made of partitions, and each partition is an append-only log with its own offsets starting at 0.
• The producer picks the partition; with a key, it hashes the key, so one key always goes to one partition.
• Order is promised only inside a partition, never between partitions or between topics.
• Retention, cleanup policy, partition count, and readers belong to the topic, so events in one topic share them.
• Events that must be read in order relative to each other must share a topic and a key.
• Separate a stream when its rate, retention, key, or readers differ, as courier locations differ from order events.

The courier locations above arrive at 2,250 events a second, and the partition count decides how many machines and readers can share that load. The next lesson, How Many Partitions, works out that count from the data rate and shows what happens when it has to change.

Practice Questions

Try each question first, then open the answer.

1. A topic has four partitions. A consumer reads offset 5 of partition 0 and then offset 5 of partition 2. Are these the same event, and which one was written first?

<details> <summary>Show answer</summary>

They are different events, and the offsets say nothing about which came first. Each partition numbers its own events from 0, so offset 5 exists separately in every partition. Kafka orders events only inside one partition, so comparing offsets across partitions is meaningless.

</details>

2. A payments platform publishes PaymentAuthorized and PaymentCaptured to two separate topics. A ledger service reads both and sometimes sees a capture before its authorization. Why, and what would fix it?

<details> <summary>Show answer</summary>

Order exists only inside a partition, and the two events are in different topics. The ledger reads each topic independently, so nothing orders a capture after its authorization. Publishing both types to one topic, keyed by payment ID, puts each payment's events in one partition, in the order they were written.

</details>

3. A large stream carries 600 events a second of about 1 KB each and needs 2 days of history. A small stream carries 20 events a second of about 2 KB each and needs 90 days. With three copies of every event, about how much disk does one shared topic kept for 90 days need, compared with two separate topics?

<details> <summary>Show answer</summary>

About 14.9 TB shared, against about 1.2 TB as two topics. The large stream writes about 600 KB a second, or about 51.8 GB a day; the small one writes about 40 KB a second, or about 3.5 GB a day. Shared, 90 days x 3 copies of both is about 55.3 GB x 270, or about 14.9 TB. Split, the large stream needs 51.8 x 2 x 3, about 311 GB, and the small one 3.46 x 90 x 3, about 930 GB, for about 1.2 TB in total.

</details>

4. In the sample run, orders D-6624 and D-6627 share partition 0. Does that break the order of either order's events?

<details> <summary>Show answer</summary>

No. Their events alternate in the partition, but each order's own four events appear in the order they were written, at offsets 0, 2, 4, 6 and 1, 3, 5, 7. A reader that tracks each order by its key sees every order's steps in sequence.

</details>

5. A team proposes one topic per event type for a shopping cart: ItemAdded, ItemRemoved, and CartCheckedOut. The checkout service rebuilds each cart from these events. What goes wrong, and what layout fits better?

<details> <summary>Show answer</summary>

The checkout service can apply events out of order. An ItemRemoved read from its own topic before the matching ItemAdded leaves the rebuilt cart wrong. All three types describe one cart and must be read in sequence, so they belong in one topic keyed by cart ID.

</details>