What Events Cost

A network of 40 clinics books 24,000 appointments a day, almost all of them between 08:00 and 20:00. That is 2,000 bookings an hour, or about 33 a minute. Each booking matters to three other services: the reminder sender texts the patient the day before, the calendar sync puts the appointment in the doctor's calendar, and the no-show analytics service counts missed visits per clinic.

The booking service can call those three services directly, or it can publish one AppointmentBooked event and let each service read it. The event design removes three calls from the booking path, and it adds four costs: delay, duplicates, harder debugging, and a broker to operate.

Each of those costs can be measured with the clinic's own numbers. Measuring them first makes the decision a comparison rather than a preference.

The Two Designs

In the direct design, the booking service saves the appointment and then calls the reminder sender, the calendar sync, and the analytics service, one after another. If any call fails, the booking service must decide whether to fail the booking, retry, or skip that service.

In the event design, the booking service saves the appointment and publishes AppointmentBooked to a topic named appointment-events, with the appointment ID as the key. Each of the three services reads the topic under its own group name, at its own pace. The booking service never learns whether they succeeded.

Everything below compares these two designs on the same 33 bookings a minute.

Delay

Delay is the time between an event being stored and a consumer acting on it. In normal running it is small. A Kafka producer waits up to linger.ms to group records into one request, and the default printed by the producer in a real run was 5 ms. A consumer that keeps up then reads each event shortly after it is stored.

The large delays come from a consumer that falls behind. Here is a simple example. The calendar sync is down for 30 minutes, and bookings keep arriving at about 33 a minute. When it returns, 1,000 events are waiting for it. If each calendar write takes 50 ms, the service works through 20 events a second, so it needs about 50 seconds to catch up.

During those 30 minutes, the doctor's calendar shows slots as free that are already booked. If the front desk checks availability in the calendar, two patients can be booked into the same slot. The design rule that follows is plain: decide anything that must be correct now, such as whether a slot is free, from the service that owns the data, never from a copy updated by events.

The direct design does not avoid this cost; it moves it. With direct calls, the same outage either fails about 1,000 bookings or forces the booking service to store and retry those calls itself.

Duplicates

A duplicate is the same event processed more than once by the same consumer. Kafka's documentation states that it "guarantees at-least-once delivery by default." The reason sits in how a consumer works: it processes records and saves its position separately. Its position is "the offset of the next message to consume," and if the process crashes after processing records but before saving that position, its replacement starts from the last saved position and processes those records again.

The program below shows this with the reminder sender. The first process sends four reminders, saves its position as 4, sends three more, and then stops without saving, as a crash would. A replacement process in the same group then takes over.

for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
    remindersSent.merge(r.key(), 1, Integer::sum);
    System.out.println(name + ": reminder sent for " + r.key() + " (offset " + r.offset() + ")");
    if (r.offset() == commitAt) {
        // The saved position is the offset of the NEXT record to read.
        consumer.commitSync(Map.of(new TopicPartition(TOPIC, 0), new OffsetAndMetadata(r.offset() + 1)));
        System.out.println(name + ": position saved as " + (r.offset() + 1));
    }
    if (r.offset() == crashAt) return;
}
Copy

Running it against a single Kafka 4.4 broker printed the following.

Output:

first process: reminder sent for APT-3301 (offset 0)
first process: reminder sent for APT-3302 (offset 1)
first process: reminder sent for APT-3303 (offset 2)
first process: reminder sent for APT-3304 (offset 3)
first process: position saved as 4
first process: reminder sent for APT-3305 (offset 4)
first process: reminder sent for APT-3306 (offset 5)
first process: reminder sent for APT-3307 (offset 6)
replacement: reminder sent for APT-3305 (offset 4)
replacement: reminder sent for APT-3306 (offset 5)
replacement: reminder sent for APT-3307 (offset 6)
replacement: reminder sent for APT-3308 (offset 7)
replacement: reminder sent for APT-3309 (offset 8)
replacement: reminder sent for APT-3310 (offset 9)
replacement: position saved as 10
APT-3305 got 2 reminders
APT-3306 got 2 reminders
APT-3307 got 2 reminders
Copy

Three patients received two reminders each. Nothing went wrong inside Kafka: the replacement started from offset 4, which is exactly the position that was saved.

The size of the window depends on how often the position is saved. By default a consumer saves it automatically on a timer, and auto.commit.interval.ms defaults to 5,000 ms. A crash can therefore repeat roughly the last 5 seconds of work. If the reminder sender handles 40 texts a second during its evening run, one crash can send a second reminder to about 200 patients.

Whether a duplicate matters depends on what the consumer does with the event:

Every consumer of an event must be checked for what a repeat does. The direct design has repeats too, because a retried call can succeed twice, but events make them a routine part of normal operation.

Debugging

A patient calls to say that no reminder arrived for appointment APT-3304. In the direct design, the booking service's log holds the call to the reminder sender and its response, so one log answers the question.

In the event design, the answer can be in any of four places:

This search is only practical when every service logs the same identifier. Using the appointment ID as the event key helps, and an event can also carry headers, which are optional metadata attached to the event. A trace ID, one identifier copied into a header and into every log line about the same action, lets one search find every step that action passed through.

Running the Broker

The storage cost for the clinic is small. Suppose each event is about 1 KB. Then 24,000 events take about 24 MB a day, and Kafka's default retention of 7 days keeps about 168 MB. The documented durable setup uses a replication factor of 3, which stores every event three times, so the total is about 504 MB.

The real cost is the cluster itself. A replication factor of 3 needs at least three brokers, and Kafka's documentation says that running the broker and controller in one process should be avoided in critical deployments. Someone must monitor these machines, upgrade them, and respond when they fail. That cost is nearly the same whether the clinic publishes 24,000 events a day or 24 million, so it weighs most heavily on small systems. A managed Kafka service moves this work to a provider, at a price.

When the Costs Are Worth Paying

For this clinic, events fit: three readers, a fourth planned, and no reader whose result the booking needs. The decision changes when there is only one reader. If the only follow-up to a booking were the reminder, a direct call with a retry, or a database table that a scheduled job reads, would give the same result without a cluster to operate.

Using This in an Interview

When you propose Kafka, name its costs before anyone asks: "Each reader may lag behind, so anything that must be current, such as slot availability, is read from the booking service. Delivery is at least once, so the reminder sender records which appointment IDs it has already handled."

Two follow-up questions are common:

• "What happens if the calendar sync is down for half an hour?" About 1,000 events wait in the topic. When the service returns, it catches up in under a minute, and nothing is lost.
• "Why not call the three services directly?" Direct calls make every booking depend on all three services being up, and a fourth service means changing the booking code. With one reader and no Kafka in place, a direct call would be the better choice.

Key Takeaways

• Events trade three direct calls for four costs: delay, duplicates, harder debugging, and a broker to operate.
• Delay is small while consumers keep up and large while one is behind: 30 minutes down at 33 events a minute leaves 1,000 events waiting.
• Never decide something that must be correct now from a copy that events keep updated.
• Kafka delivers at least once by default. A crash between doing the work and saving the position repeats that work, about 5 seconds of it with the default auto commit.
• A shared identifier in keys, headers, and logs is what makes an event flow possible to debug.
• The cluster is a mostly fixed cost, so events become worth their cost sooner with more readers and higher volume.

Delay and repeats both come from a property of Kafka that the previous lesson only mentioned: events stay stored after they are read, and each reader keeps its own position. Message queues behave differently, and the next lesson, Queues and Logs, compares the two.

Practice Questions

Try each question first, then open the answer.

1. A payments service publishes PaymentReceived at 50 events a minute. Its ledger consumer is down for 40 minutes, and once it returns it processes 25 events a second. How many events are waiting, and about how long does it take to catch up?

<details> <summary>Show answer</summary>

2,000 events and about 80 seconds. The backlog is 50 x 40 = 2,000 events. At 25 events a second the consumer needs 2,000 / 25 = 80 seconds, plus a little more for the roughly 67 events that arrive during those 80 seconds.

</details>

2. A consumer uses the default auto commit and processes 120 events a second. About how many events can a single crash process a second time, and why?

<details> <summary>Show answer</summary>

About 600. The position is saved every 5,000 ms by default, so a crash can lose up to about 5 seconds of saved progress. The replacement repeats that work: 120 x 5 = 600 events, sometimes slightly more.

</details>

3. A store shows "in stock" on product pages using a copy of inventory that an event consumer keeps updated. Checkout also reads that copy to decide whether an order can be accepted. What is wrong?

<details> <summary>Show answer</summary>

Checkout makes a must-be-correct decision from a copy that can lag. While the consumer is behind, the copy can show stock that has already been sold, so orders are accepted that cannot be filled. Product pages may use the copy, but checkout should ask the inventory service that owns the data.

</details>

4. A topic receives 300,000 events a day of about 2 KB each, kept for 7 days with a replication factor of 3. About how much disk does it use across the cluster?

<details> <summary>Show answer</summary>

About 12.6 GB. One day is 300,000 x 2 KB = 600 MB. Seven days is 4.2 GB, and three copies make about 12.6 GB.

</details>

5. A new consumer increments a "bookings today" counter by one for every AppointmentBooked event. Why will the number drift upward over time, and how can it be fixed?

<details> <summary>Show answer</summary>

Every repeated event adds one more. Delivery is at least once, so each crash or restart that repeats events inflates the counter. Counting distinct appointment IDs instead of incrementing makes a repeat harmless, because the same ID is only counted once.

</details>