Course Introduction

This course prepares you to make the decisions that an event-driven system depends on. Should this connection be a request or an event? How many partitions does a topic need, and what should its key be? How many copies of each event must exist before a write counts as done, and how long should a topic keep its data? Should duplicates be prevented, or should every consumer be built to tolerate them? Which broker fits the workload: Kafka, RabbitMQ, Amazon SQS, or Apache Pulsar?

Each of those questions has a cost on both sides, and the course treats them as decisions to be argued with numbers rather than as features to memorize. Every lesson takes one decision, works it with real load figures and real Kafka behavior, states what each option costs, and says when the other option is the better one.

The course is not a tour of Kafka's settings. It uses Kafka 4.4 as its main broker because Kafka shows most of the trade-offs clearly, and it compares Kafka with the other brokers wherever the choice of broker is itself the decision.

Who This Course Is For

The course is written for a working backend engineer who builds HTTP services and works with databases, but who has not yet designed a system around Kafka. You do not need to have run a broker before. Every Kafka term is described in plain words the first time it appears, and then used freely.

It also suits an engineer who already runs Kafka and wants the reasons behind its settings. Many production problems come from a default that nobody chose, such as an automatic commit that repeats work after a crash, or a key that sends most of the traffic to one partition. The lessons show where those defaults come from and what each alternative costs.

What Each Chapter Decides

The chapters move from whether to use events at all, through the parts of Kafka that each design decision depends on, to complete designs.

• Should This System Use Events? Which calls need an answer now and which can be announced, what events cost in delay and duplicates, and when a design is better without them.
• Topics, Partitions, and Order. How to group events into topics, how many partitions a topic needs, which key to choose, and what order you can actually rely on.
• Consumers and Scaling. How consumer groups divide work, how rebalancing behaves under the classic and the newer protocol, and when a share group fits better than a consumer group.
• Durability and Storage. Replication and acknowledgments for a stated tolerance for loss, what happens when brokers fail, retention and compaction with storage arithmetic, safe replays, and the controller.
• Duplicates and Exactly Once. Where repeats come from, how consumers tolerate them, what the idempotent producer and transactions really guarantee, and how to retry without blocking a partition.
• Consistency, Schema Changes, and Lag. The outbox and sagas on Kafka, changing an event's schema on a topic that lives for years, and measuring and diagnosing consumer lag.
• Choosing a Broker. A set of questions that decide the choice before any product is named, then Kafka compared with RabbitMQ, Amazon SQS, and Apache Pulsar on one workload.
• Design: An Order Pipeline, A Notification System, and An Activity Feed. Three complete designs, each stated at topic level: topics, keys, partition counts, replication, retention, consumer groups, failure walks, and alerts.

The early chapters each settle one kind of decision. The three design chapters combine them, and they link back to the lesson where each decision was first made instead of teaching it again.

How a Lesson Is Built

Lessons share the same parts. Each one opens in the way that suits its decision.

1. One decision. A lesson may open with a definition, a direct question, a set of numbers, real output from a run, or a short incident, and then it states the one question it answers.
2. A running example. Most lessons follow a single system, such as a parcel tracking service or a clinic's appointment reminders, from start to end.
3. Real numbers. Load, storage, and lag are worked out with stated assumptions, for example "suppose each event is about 1 KB," so you can repeat the arithmetic with your own figures.
4. Real runs. Kafka behavior is shown with Java programs that ran against a real Kafka 4.4 broker. The output in the lesson is the output that the run printed, and a check confirms that the two match.
5. Costs on both sides. Each option is stated with its cost, and with the condition under which the other option is better.

Each lesson ends with a short list of key takeaways and with practice questions whose answers are folded, so you can try each one before reading the answer. Most lessons also include a short section on explaining the decision in an interview. Each chapter closes with a quiz.

Claims about Kafka come from the Apache Kafka documentation for version 4.4. Claims about RabbitMQ, Amazon SQS, Apache Pulsar, and schema tools come from their official documentation, and settings for those products are quoted from that documentation rather than shown as program output.

Running the Code

The course player does not run Kafka code. A Kafka program needs a running broker, and the player has none, so every sample appears as code together with the output it printed in a real run. To run a sample yourself, start a broker on your own machine.

The Kafka documentation gives a single command that starts a broker in Docker:

docker run -p 9092:9092 apache/kafka:4.4.0
Copy

The steps below were checked on Linux against this course's first sample. They also give you Kafka's client libraries without a separate download. Wait a few seconds after the first command, until the broker answers, before creating the topic. The loop on the second line does that waiting for you.

docker run -d --name kafka -p 9092:9092 apache/kafka:4.4.0
until docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list >/dev/null 2>&1; do sleep 1; done
docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic parcel-events --partitions 1 --replication-factor 1
docker cp kafka:/opt/kafka/libs ./klibs
java -cp "klibs/*" ParcelScan.java
Copy

The first line starts the broker in the background under the name kafka. The second waits until the broker answers, and the third creates the topic that the sample writes to. The fourth copies the client libraries out of the container, and the last runs the sample as a single file with the Java source launcher. Kafka 4.4 fully supports Java 17, 21, and 25.

Each lesson that has a sample names the topics it uses, so you change only the topic line and the file name. Samples that delete or compact data were run on Linux. Kafka's documentation says that Windows is not a well supported platform, so on Windows, run the broker in Docker as shown rather than directly.

The course starts with the first decision of all: whether a connection between two services should be a request or an event. That decision is the subject of Requests and Events.