Queues and Logs

On This Page

Queues

Logs

Both Jobs on One Topic

Comparing Queues and Logs

The Price of Keeping Events

Choosing Between Them

Using This in an Interview

Key Takeaways

Practice Questions

An accounting service publishes an InvoiceRequested message for every completed order, about 12,000 a day. Two jobs depend on those messages. The invoice generator must turn each one into a PDF and email it exactly once, and after that the message is of no further use to it. The audit team needs the opposite: every message kept, in order, so that it can answer a question such as "which invoices were requested for customer C-882 last month?"

The same stream of messages therefore has two very different futures. For the first job, a handled message should disappear. For the second, a handled message is the whole point.

Messaging systems are built around one of these two futures or the other. This page defines both, runs both jobs against the same Kafka topic, and gives the question that decides between them.

Queues

A queue holds each message until one consumer has handled it, and then removes it. RabbitMQ's documentation describes its queues as having "destructive consume behaviour": messages "are deleted from the queue when a consumer is finished with them," so consumed messages cannot be read again. Amazon SQS works the same way in effect: a consumer receives a message, and "after processing, you must explicitly send a delete request to remove the message permanently."

A queue is built for sharing work. Several workers read from one queue, and each message goes to one of them. While a worker holds an SQS message, the message is hidden from the other workers, by default for 30 seconds, and it becomes visible again if the worker does not delete it in time.

Here is a simple example. Generating one invoice PDF takes 400 ms, so one worker handles 2.5 invoices a second, or 9,000 an hour. On a normal day the accounting service requests its 12,000 invoices over about eight working hours, or about 1,500 an hour, which one worker handles easily. At month end the rate rises to about 4 a second, so a second worker is added, and the two together handle 5 a second. Nothing else changes: the queue simply hands each message to whichever worker is free.

Benefit: simple work sharing. Adding workers adds capacity, and no worker needs to know about the others.
Benefit: nothing to clean up. Handled messages leave the queue, so storage holds only work that is still waiting.
Cost: no history. Once a message is handled it is gone. A new reader cannot see past messages, and a bug cannot be fixed by processing old messages again.

Logs

A log is an ordered sequence of events that are kept after they are read, where each reader tracks its own position. Kafka topics are logs. Kafka's documentation states that "events are not deleted after consumption"; instead, each topic keeps events for a configured time, 7 days by default. A reader's position is an offset, and the documentation adds that a consumer "can deliberately rewind back to an old offset and re-consume data."

Because nothing is removed by reading, any number of readers can use one log, each at its own pace. A new reader can start from the oldest stored event and build its own view of the history.

Both Jobs on One Topic

The program below publishes six InvoiceRequested events to a topic named invoice-events. It then reads the topic under different group names, saving each group's position after it reads. The last step rewinds the invoice worker to the oldest event, using the method that Kafka provides for rewinding.

report("invoice-worker, first run", read("invoice-worker", false, 6));
report("invoice-worker, restarted", read("invoice-worker", false, 0));
report("audit-history", read("audit-history", false, 6));
report("audit-rebuild, a new group", read("audit-rebuild", false, 6));
report("invoice-worker, rewound", read("invoice-worker", true, 6));

// Rewinding moves this group's position back to the oldest stored event.
if (rewind) consumer.seekToBeginning(parts);

Running it against a single Kafka 4.4 broker printed the following.

Output:

invoice-worker, first run: read 6 events, offsets 0 to 5
invoice-worker, restarted: read 0 events
audit-history: read 6 events, offsets 0 to 5
audit-rebuild, a new group: read 6 events, offsets 0 to 5
invoice-worker, rewound: read 6 events, offsets 0 to 5

Each line shows one property of a log:

The invoice worker behaves like a queue consumer. After a restart it reads nothing, because its saved position already points past the six events.
The audit history reads everything, even though the invoice worker had already handled those events.
A brand-new group starts from offset 0 when it is set to start from the oldest stored event, as every group in this program is. This is how an audit table or a search index can be rebuilt from the stored events.
Rewinding repeats side effects. The rewound invoice worker read all six events again. If it emailed invoices, six customers would receive a second copy.

A queue removes, a log keeps

Replaying a log is safe only for readers whose work can be repeated. Rebuilding an audit table from offset 0 produces the same table again. Re-sending invoices does not produce the same result, because the customer receives each email twice.

Comparing Queues and Logs

	Queue	Log
After a message is handled	It is removed	It stays until the retention time ends
Several readers	Share the work; each message goes to one	Each reads every event at its own pace
Adding a new reader	It sees only new messages	It can start from the oldest stored event
Reprocessing old messages	Not possible	Rewind the position, or start a new group
Storage	Only waiting messages	Every event for the whole retention time
Examples	RabbitMQ queues, Amazon SQS	Kafka topics, RabbitMQ streams

Products built around one model also offer the other. RabbitMQ offers streams, and its documentation says that "as streams never delete any messages, any consumer can start reading/consuming from any point in the log." Kafka offers share groups, which it describes as production-ready since Kafka 4.2 and suited to "cases where records are processed one at a time, rather than as part of an ordered stream." The choice is therefore between two models of reading, not only between two products.

The Price of Keeping Events

A log is not free history, because every kept event takes disk space. Here is a simple example. The audit team wants 90 days of invoice events. Each event is about 2 KB, so 12,000 events a day take about 24 MB. Ninety days take about 2,160 MB, and the documented durable setup keeps three copies of every event, which brings the total to about 6.5 GB.

That is a modest amount, but the arithmetic must be done for every topic, because the same 90 days for a topic with 100 times the traffic needs about 650 GB. Retention is a per-topic decision: the invoice topic can keep 90 days while a high-volume topic keeps 1 day.

Choosing Between Them

Ask one question about the messages: will anyone need a message again after it has been handled?

No, the work only needs to be done once: a queue fits. Sending invoice emails, resizing uploaded images, and charging a card for a confirmed order are all work that is finished once it succeeds.
Yes, now or possibly later: a log fits. Audit history, a search index that may need rebuilding, and a new service that needs past events all depend on messages staying readable.
Both, as with invoices: a log can serve both jobs, with one group doing the work once and other groups reading the history. The worker that sends emails must then never be rewound, or it must be able to recognize invoices it has already sent.

Using This in an Interview

State which property the design depends on: "Invoice generation needs each request handled once, so a queue would be enough. But audit needs the full history and we may add readers later, so I would use a Kafka topic with 90 days of retention, one consumer group for invoice generation and one for audit."

Two follow-up questions are common:

"How would you rebuild the audit table after a bug?" Start a new consumer group from the oldest stored event and write a fresh table. Rebuilding works only within the retention time.
"Could you use RabbitMQ instead?" Yes for the invoice work, using a queue. For the audit history, a RabbitMQ stream or a separate audit database would be needed, because a queue does not keep a message after it is consumed.

Key Takeaways

A queue removes each message once a consumer has handled it, so it shares work well and keeps no history.
A log keeps events after they are read, and each reader keeps its own position, so readers are independent and history can be replayed.
In the real run, a restarted worker read nothing new, while a brand-new group read all six events from offset 0.
Replay is safe only for readers whose work can be repeated; rewinding a reader that sends emails sends them again.
A log's history costs disk: events per day x size x days kept x copies, worked out per topic.
Decide with one question: will anyone need a message again after it has been handled?

The invoice events in the program above were short strings. A real event needs a name, a key, identifiers, and a decision about how much data to carry, and the next lesson, Designing an Event, works through those choices.

Practice Questions

Try each question first, then open the answer.

1. A video service must create a thumbnail for every uploaded video, once, and nothing else reads the upload messages. Which model fits, and why?