Interview Bootcamp
Vote

0% completed

​

Kafka Workflow

Kafka offers two different ways to deliver messages. We walk through both next.

Kafka can deliver messages two ways: as pub-sub messaging and as queue-based messaging. Both are fast, reliable, persisted to disk, fault-tolerant, and able to run with zero downtime. A producer sends its message to a topic the same way in each case. What changes is how consumers receive it, depending on which model they pick. The steps below walk through both.

Kafka workflow as pub-sub messaging

Here is the pub-sub workflow, step by step:

  • Producers publish messages to a topic.

.....

.....

.....

Like the course? Get enrolled and start learning!
C

consumerkey

· 6 months ago

Upon receiving the acknowledgment, Kafka increments the offset and updates it in the ZooKeeper. Since offsets are maintained in the ZooKeeper, the consumer can read the next message correctly, even during broker outages.

That part contradicts "dumb broker, smart consumer" stated before and does not reflect how modern Kafka (version 0.9 +) works. So be aware that offset storage in Zookepeer is a legacy approach, currently replaced by special internal topic __consumer_offset .

J

Junaid Effendi

· 4 years ago

Some questions for everyone to google for answers to understand and clarify:

  • When a consumer wants to remind from a X offset, how would consumer know what offset represents what data?
  • Data is stored on disk and never deleted, so the rewind is possible and offset remains the same for the same data, correct?
  • What if data size grows and clean up is required on disk?
  • Does kafka uses cache?
  • How does kafka writes on disk, like is it a plain file?

Reading Progress

0%

Vote for new content