Grokking the System Design Interview, Volume II
Vote

0% completed

​

Kafka: Deep Dive

We have covered Kafka's core ideas. Now we go deeper: how topics split into partitions, and how leaders and followers keep each partition safe.

Kafka is a collection of topics. A topic can grow large, so Kafka splits it into partitions: smaller pieces that perform better and scale further.

Topic partitions

A topic is spread across a number of partitions, and each partition can live on a separate Kafka broker. A new message arriving on a topic gets appended to one of that topic's partitions. The producer decides which partition a message goes to, based on the message's data

.....

.....

.....

Like the course? Get enrolled and start learning!
J

Junaid Effendi

· 4 years ago

What happens if we remove a kafka broker/server? How the data is rebalanced? What happens if we add new server/partition, how is the new data assigned to new partition?

C

Camoen V

· 4 years ago

I was curious about what logical reasons a Producer would have for publishing to a single partition, rather than a single Topic. The example provided at https://codingharbour.com/apache-kafka/the-introduction-to-kafka-topics-and-partitions/ was helpful ("The link between a record and a partition").

Partitions maintain ordering, which is important for event ingestion in some scenarios. For example, if a user buys an item and then cancels their purchase, we want to ensure that these events are in order. In this case, the producer could use customerID as the key of the Kafka record, to ensure that both events go to the same partition (and then are consumed in order by the 'event consumer').

A

Amey Naik

· 5 years ago

"Ordering of messages is maintained at the partition level, not across the topic."

Show 1 reply

Reading Progress

0%

Vote for new content