Grokking the System Design Interview, Volume II
Vote

0% completed

​

Kafka: Deep Dive

As of now, we have discussed the core concepts of Kafka. Let us now throw some light on the workflow of Kafka.

Kafka is simply a collection of topics. As topics can get quite big, they are split into partitions of a smaller size for better performance and scalability.

Topic partitions

Kafka topics are partitioned, meaning a topic is spread over a number of 'fragments'. Each partition can be placed on a separate Kafka broker. When a new message is published on a topic, it gets appended to one of the topic's partitions

.....

.....

.....

Like the course? Get enrolled and start learning!
C

Camoen V

· 4 years ago

I was curious about what logical reasons a Producer would have for publishing to a single partition, rather than a single Topic. The example provided at https://codingharbour.com/apache-kafka/the-introduction-to-kafka-topics-and-partitions/ was helpful ("The link between a record and a partition").

Partitions maintain ordering, which is important for event ingestion in some scenarios. For example, if a user buys an item and then cancels their purchase, we want to ensure that these events are in order. In this case, the producer could use customerID as the key of the Kafka record, to ensure that both events go to the same partition (and then are consumed in order by the 'event consumer').

A

Amey Naik

· 5 years ago

"Ordering of messages is maintained at the partition level, not across the topic."

Show 1 reply
J

Junaid Effendi

· 4 years ago

What happens if we remove a kafka broker/server? How the data is rebalanced? What happens if we add new server/partition, how is the new data assigned to new partition?