Grokking the System Design Interview, Volume II
Vote

0% completed

​

Kafka Workflow

Let's explore different messaging workflows that Kafka offers.

Kafka provides both pub-sub and queue-based messaging systems in a fast, reliable, persisted, fault-tolerance, and zero downtime manner. In both cases, producers simply send the message to a topic, and consumers can choose any one type of messaging system depending on their need. Let us follow the steps in the next section, to understand how the consumer can choose the messaging system of their choice.

Kafka workflow as pub-sub messaging

Following is the stepwise workflow of the Pub-Sub Messaging:

.....

.....

.....

Like the course? Get enrolled and start learning!
J

Junaid Effendi

· 4 years ago

Some questions for everyone to google for answers to understand and clarify:

  • When a consumer wants to remind from a X offset, how would consumer know what offset represents what data?
  • Data is stored on disk and never deleted, so the rewind is possible and offset remains the same for the same data, correct?
  • What if data size grows and clean up is required on disk?
  • Does kafka uses cache?
  • How does kafka writes on disk, like is it a plain file?
C

consumerkey

· 4 months ago

Upon receiving the acknowledgment, Kafka increments the offset and updates it in the ZooKeeper. Since offsets are maintained in the ZooKeeper, the consumer can read the next message correctly, even during broker outages.

That part contradicts "dumb broker, smart consumer" stated before and does not reflect how modern Kafka (version 0.9 +) works. So be aware that offset storage in Zookepeer is a legacy approach, currently replaced by special internal topic __consumer_offset .