Grokking Kafka and Event-Driven Architecture
Learn to design event-driven systems with Apache Kafka: when to use events, how to size topics and partitions, and how to reason about durability, duplicates, ordering, and consumer lag.
Course Overview
This course teaches event-driven architecture through the decisions an engineer makes when building it. Each lesson takes one decision, such as whether a call should be a request or an event, how many partitions a topic needs, or which acknowledgment setting a workload can afford, and works it through with real numbers, the cost of each option, and the case where the other option is better. Every code sample was run against a real Kafka 4.4 broker, and each lesson shows the output it produced. The course starts with whether a system should use events at all. It then covers topics, partitions, and keys; consumer groups and share groups; replication, retention, and compaction; duplicate messages and exactly-once processing; and the outbox pattern, sagas, schema changes, and consumer lag. It compares Kafka with RabbitMQ, Amazon SQS, and Apache Pulsar, and ends with complete designs of an order pipeline, a notification system, and an activity feed. Every chapter lesson ends with practice questions, every chapter ends with an assessment, and most lessons close with a short section on explaining the decision in a system design interview.
What you'll learn in Grokking Kafka and Event-Driven Architecture
- Decide when a call should be a request and when it should be an event, and price what events cost in delay and repeats.
- Size a topic's partitions from its write and read rates, and pick keys that keep order without creating hot partitions.
- Set replication, acks, and min.insync.replicas for a stated loss tolerance, and read what happens when brokers fail.
- Trace where duplicate messages come from, build consumers that tolerate repeats, and know what transactions cover.
- Keep services consistent with the outbox pattern and sagas, with the right Kafka keys for each.
- Measure consumer lag in records and in time, and find its cause from real tool output.
- Apply every decision in complete designs: an order pipeline, a notification system, and an activity feed.
- Choose between a queue and a log, and design events with clear names, keys, identifiers, and the right amount of data.
- Scale consumers with consumer groups and share groups, and stop slow handlers from causing repeated rebalances.
- Plan retention and compaction with storage arithmetic, and replay events without repeating side effects.
- Design retry topics and dead letters so that one failing message does not block a whole partition.
- Change event schemas safely, including on compacted topics that keep old records for a long time.
- Choose between Kafka, RabbitMQ, Amazon SQS, and Apache Pulsar for a stated reason.
Course Content
Should This System Use Events?
Topics, Partitions, and Order
Consumers and Scaling
Consistency, Schema Changes, and Lag
What people say about our courses
About the Author
Arslan Ahmad
Industry Expertise & Leadership
Arslan Ahmad is the lead author of Grokking Kafka and Event-Driven Architecture. As the founder of Design Gurus and a former FAANG hiring manager, he has worked at industry giants like Facebook (now Meta) and Microsoft.
He has conducted hundreds of system design interviews, giving him unique insight into what top tech companies look for in candidates.
The course also incorporates expertise from senior engineers at Google, Meta, Amazon, Microsoft, and Uber, ensuring you learn system design best practices from professionals who have built and scaled real-world systems.
500+
Interviews Conducted
10k+
Students Taught
Related Courses
$148
$72
$98
FAQs
What is Grokking Kafka and Event-Driven Architecture?
It is a text course on designing event-driven systems with Apache Kafka. Each lesson takes one design decision, such as whether a call should be an event, how many partitions a topic needs, or which acknowledgment setting a workload can afford, and works it through with real numbers, real Kafka behavior, and the cost of each option. The course ends with complete designs of an order pipeline, a notification system, and an activity feed.
Who is this course for?
It is for backend engineers who build services and want to design with Kafka well, for engineers who already run Kafka and want to understand the trade-offs behind its settings, and for engineers preparing for system design interviews where events and message brokers come up.
What should I know before starting?
You should be comfortable building backend services that talk over HTTP and store data in a database, and you should be able to read short Java programs. You do not need any experience with Kafka or other message brokers, because the course defines every term when it first appears.
Is this a Kafka tutorial?
No. The course teaches the decisions an engineer makes when designing with events, and it uses Kafka to show each one in practice. It shows the client code and settings that a decision needs, but it does not walk through every Kafka API or command-line option.
Which version of Kafka does the course use?
Kafka 4.4. It covers current Kafka, including KRaft, which replaced ZooKeeper, the newer consumer rebalance protocol, share groups for queue-style reading, and eligible leader replicas, and it says when a behavior changed in a recent version.
Do I need to install Kafka?
Not to follow the course. Every code sample was run against a real Kafka 4.4 broker, and each lesson shows the code next to the output it produced, because the course player cannot run a Kafka broker. To run the samples yourself, start a broker with docker run -p 9092:9092 apache/kafka:4.4.0; the Course Introduction gives the exact steps.
Which programming language does the course use?
The samples use Kafka's own Java client, and the design lessons use short pseudocode. The decisions themselves apply to any language that has a Kafka client.
Does the course compare Kafka with other message brokers?
Yes. One chapter compares Kafka with RabbitMQ, Amazon SQS, and Apache Pulsar on one shared workload, using each product's official documentation, and ends with a way to choose a broker for a stated reason. It does not use vendor benchmarks.
Is there any practice, or is it only reading?
Every lesson after the introduction ends with practice questions whose answers you can open after trying them, and every chapter after the introduction ends with an adaptive assessment. Many of the practice questions are short design calculations, such as sizing partitions or storage.
Will this course help with system design interviews?
Most lessons end with a short section on how to explain the decision in an interview, with likely follow-up questions, and the three complete designs are common interview questions. The rest of each lesson teaches the subject itself, so the same material applies to building production systems.
Is the course text-based or video-based?
It is text-based, with animated diagrams that show events moving through each design, so you can read at your own pace and return to any lesson as a reference.
