Kafka vs RabbitMQ: Log vs Queue, and How to Choose

Choose Kafka when many independent consumers need the same stream of events, or when you need to read old events again. Choose RabbitMQ when each message is a task that one worker should handle once, especially when messages need flexible routing, priorities, or low latency at moderate volume. Most other differences between the two systems follow from that single split.

The split exists because the two systems store messages in different ways. Kafka writes every message to a log on disk and keeps it for a set period, whether anyone has read it or not. RabbitMQ holds a message in a queue only until a consumer confirms it, and then deletes it.

This article explains the two models, compares them on retention, ordering, routing, delivery guarantees, throughput, and operations, and ends with a checklist and three worked examples.

What is the difference between a log and a queue?

A log is a file that only grows at the end. New messages are appended, nothing is changed in place, and every message gets a sequence number called an offset. A reader keeps its own position in the log, so two readers can be at different points in the same data at the same time. Kafka is built around this structure (a distributed commit log).

A queue is a waiting line. A message enters at the back, one consumer takes it from the front, and once that consumer confirms it, the message is gone. The broker, which is the server that stores and forwards messages, tracks which messages are still waiting and which are being worked on. RabbitMQ is built around this structure (a message broker with queues).

The practical difference is who remembers progress. In Kafka, each consumer remembers how far it has read, and the broker only stores data. In RabbitMQ, the broker remembers what has been delivered and acknowledged, and the consumer only receives work. This one design choice explains why Kafka is good at replay and fan-out, and why RabbitMQ is good at task distribution and per-message control.

How does Kafka store and deliver messages?

Kafka groups messages into topics, such as orders or page-views. Each topic is split into partitions, which are separate logs that can live on different servers. A producer, the program that sends messages, chooses a partition for each message, usually by hashing a key such as the customer ID.

Partitions are the unit of parallelism. If a topic has 12 partitions, up to 12 consumers in one group can read it at the same time, each owning one or more partitions. A consumer group is a set of consumers that share the work of reading a topic, and Kafka assigns each partition to exactly one member of the group.

Several groups can read the same topic without affecting each other. A billing service, a search indexer, and an analytics job can each form their own group, and each group keeps its own offsets. Adding a fourth reader costs nothing on the producer side, because the producer never knew about the first three.

Kafka also replicates each partition to other brokers. The usual setting is a replication factor of 3, which means every message exists on three servers. If the broker that leads a partition fails, one of the copies takes over.

How does RabbitMQ route and deliver messages?

RabbitMQ adds a routing step before the queue. A producer never sends a message straight to a queue. It sends the message to an exchange, which is a routing component that decides which queues should receive a copy.

The exchange decides by comparing the message against bindings, which are rules that connect an exchange to a queue. RabbitMQ offers four common exchange types:

• Direct: deliver to queues whose binding key equals the message's routing key, such as invoice.created.
• Topic: deliver to queues whose pattern matches the routing key, such as invoice.* or *.eu.#.
• Fanout: deliver a copy to every bound queue, ignoring the key.
• Headers: deliver based on message header values instead of the routing key.

Once a message reaches a queue, RabbitMQ pushes it to a consumer. When several consumers read from one queue, RabbitMQ spreads the messages among them. This arrangement is called competing consumers, because the consumers compete for the next message and each message goes to only one of them.

The consumer sends an acknowledgment (an ack) after it finishes the work. RabbitMQ then deletes the message. If the consumer crashes before sending the ack, RabbitMQ delivers the message to another consumer.

Retention and replay

Kafka keeps messages for a configured time or size, whether they were read or not. The default retention is seven days, and teams often set it to hours, weeks, or forever for some topics. A consumer can rewind its offset to any point inside that window and read the same events again.

Replay is useful in three common cases. A bug in a consumer corrupts its output, so you fix the bug and reprocess the last two days. A new service launches and needs the history of events to build its own database. An analytics job wants to compare this week's traffic with last week's using the same raw events.

RabbitMQ classic and quorum queues do not support replay. Once a message is acknowledged, it is deleted, and no consumer can read it again. You can keep a copy by sending it to a second queue or a database, but that copy is something you build, not something the queue provides.

Kafka has one more retention mode worth knowing. A compacted topic keeps only the latest message for each key, so a topic of customer profiles holds one current record per customer instead of every change. This makes a Kafka topic usable as a source from which a service rebuilds its current state.

Ordering guarantees

Kafka guarantees order within one partition, not across a topic. Messages with the same key go to the same partition, so all events for one customer arrive in the order they were written. Events for two different customers can be read in any relative order.

This means ordering is something you design through the key. If every event for order 8841 must be processed in sequence, use the order ID as the key. If you choose a random key, you get better spreading across partitions but lose ordering for that order.

RabbitMQ keeps messages in a queue in first-in, first-out order. That order holds while one consumer reads the queue. With several competing consumers, messages are processed in parallel, so message 2 can finish before message 1 does. A failed message that is requeued also returns to the queue after newer ones have been delivered.

RabbitMQ offers a way to keep per-key order across several consumers. The consistent hash exchange plugin sends messages with the same key to the same queue, and you run one consumer per queue. This works, but it is an extra component you configure, while in Kafka it is the default behavior of keyed partitions.

Consumer groups versus competing consumers

Both systems can spread work across many consumers, and both can deliver the same event to many services. They do it with different structures, and the difference matters when you scale.

In Kafka, parallelism inside a group is limited by the partition count. A topic with 6 partitions can use at most 6 active consumers in one group, and a seventh consumer sits idle. Adding partitions later is possible, but it changes which partition a key maps to, which can break ordering assumptions. Teams therefore choose the partition count with future growth in mind.

In RabbitMQ, you can add consumers to a queue at any time, and the broker starts sending them messages at once. The prefetch setting controls how many unacknowledged messages each consumer may hold, so a slow consumer does not collect work that a fast one could do. This makes RabbitMQ good at uneven tasks, where one job takes 50 milliseconds and the next takes 30 seconds.

Fan-out works the other way around. In Kafka, a new service adds a new consumer group, and the topic stays the same. In RabbitMQ, a new service needs its own queue bound to the exchange, and every message is then stored once per queue.

Want to practice these choices on real designs? Grokking Kafka and Event-Driven Architecture walks through partitioning, consumer groups, delivery guarantees, and broker choice with worked numbers.

Routing flexibility

RabbitMQ's exchanges let the broker make routing decisions. A payment service can publish payment.failed.eu once, and topic bindings deliver it to the European support queue, the fraud review queue, and the audit queue. No consumer has to read and discard messages it does not care about.

RabbitMQ also supports per-message features that Kafka does not offer as broker features. These include message priorities, per-message time to live, and dead-letter exchanges, which are destinations for messages that were rejected or expired. Delayed delivery is available through a plugin.

Kafka's routing is simpler: a producer chooses a topic and, through the key, a partition. A consumer that wants only some events reads the whole topic and filters them, or a stream processor writes the filtered events to a new topic. This keeps the broker fast and simple, and moves routing logic into your code or into a tool such as Kafka Streams.

Delivery guarantees

Both systems give at-least-once delivery when configured correctly. At-least-once means a message is never lost, but a consumer might receive it twice, for example after a crash between doing the work and recording progress. In both systems, the standard defense is an idempotent consumer, which is a consumer that produces the same result when it handles the same message twice.

Kafka adds two features on the producer side. An idempotent producer attaches a sequence number to each message, so the broker drops a duplicate caused by a network retry. Since Kafka 3.0, producers default to acks=all and idempotence enabled, according to the Apache Kafka 3.0 release notes, which means a write is confirmed only after all in-sync replicas have it.

Kafka transactions go one step further. A transaction lets a program read from one topic, write to another, and commit its read offset as a single unit, so either all three take effect or none do. This gives exactly-once processing for pipelines that stay inside Kafka. Once a side effect leaves Kafka, such as sending an email, you are back to at-least-once and need idempotent handling.

RabbitMQ provides publisher confirms, where the broker tells the producer that it has stored a message, and consumer acks, described earlier. Together they give at-least-once delivery. RabbitMQ has no transaction that ties a consumed message to a produced message, so exactly-once processing depends on your consumer's design.

Throughput and latency

Kafka is built for high throughput. It writes to disk in large sequential batches, lets consumers pull data in batches, and sends data from disk to the network with very little copying. A Kafka cluster commonly handles hundreds of megabytes per second, and large deployments handle much more.

RabbitMQ is built for low latency at moderate volume. It pushes each message to a consumer as soon as possible and does more work per message, such as routing and tracking acknowledgments. That extra work costs throughput, and a single queue is served by one process, which limits how fast one queue can grow.

One published benchmark shows the size of the gap. Confluent, the company founded by Kafka's creators, measured peak throughput of about 605 MB/s for Kafka and about 38 MB/s for RabbitMQ with mirrored queues. In the same test, RabbitMQ reached a p99 latency of about 1 ms at a reduced load of 30 MB/s, while Kafka's p99 was about 5 ms at 200 MB/s. The p99 latency is the time within which 99 percent of messages arrive.

Treat these numbers with care. The benchmark was run by a Kafka vendor on older versions of both systems, and RabbitMQ's mirrored queues have since been replaced by quorum queues. The direction of the result is widely accepted, however: Kafka moves far more data per second, and RabbitMQ delivers individual messages faster when the load is moderate.

For most product systems, both are fast enough. An illustrative order system with 2 million orders a day averages about 23 messages per second, with peaks perhaps ten times higher. Either broker handles that easily, so the choice should rest on the model you need, not on raw speed.

Operations: KRaft and quorum queues

Both systems changed their operating model in recent major versions, and older articles describe setups that no longer exist.

Kafka used to depend on a separate system, ZooKeeper, to store cluster metadata. Kafka now uses its own built-in consensus protocol, KRaft, and the Apache Kafka 4.0 upgrade notes state that ZooKeeper mode has been removed. A Kafka cluster is therefore one kind of server to run, not two. Operating Kafka still means planning partition counts, replication, disk space for retention, and consumer group rebalancing.

RabbitMQ used to replicate queues by mirroring classic queues across nodes. RabbitMQ 4.0 removed classic queue mirroring, according to its release notes. Replicated queues are now quorum queues, which use the Raft consensus protocol to keep copies on several nodes and stay available when a minority of nodes fail. Classic queues still exist, but only as single-node queues.

In practice, a small RabbitMQ deployment is simpler to start, and it has a management UI that many teams find easy to use. A Kafka deployment needs more planning up front, but it scales by adding brokers and partitions. Both are available as managed cloud services, which removes much of the operating work.

Are Kafka and RabbitMQ converging?

Each system has added a feature from the other's side.

RabbitMQ added streams in version 3.9. A stream is an append-only log inside RabbitMQ: consuming does not delete messages, consumers can start from an offset or a timestamp, and old data is removed by age or size. Super streams split one logical stream into partitions. This gives RabbitMQ users replay without running a second system.

Kafka added share groups through KIP-932, a design proposal also called Queues for Kafka. In a share group, consumers read the same partitions together and acknowledge records one at a time, which gives queue-like competing consumers without the one-consumer-per-partition limit. The feature was early access in Kafka 4.0, a preview in 4.1, and generally available in Apache Kafka 4.2, according to Confluent.

These features narrow the gap but do not erase it. RabbitMQ streams do not have Kafka's ecosystem of connectors and stream processing tools. Kafka share groups do not offer exchanges, priorities, or per-message routing. The core model of each system is still the best guide to which one fits.

A checklist for choosing between Kafka and RabbitMQ

Answer these questions in order. The first one that gets a clear answer usually decides.

1. Do several independent services need the same events? If yes, lean toward Kafka, because each service becomes a consumer group and the producer does not change.
2. Do you need to read old events again? Replay for bug fixes, new services, or audits points to Kafka, or to RabbitMQ streams if you already run RabbitMQ.
3. Is each message a unit of work that one worker should do once? Email sending, image resizing, and report generation point to RabbitMQ.
4. Do you need routing rules, priorities, or per-message expiry? These are built into RabbitMQ and absent from Kafka's broker.
5. What is the sustained volume? Above tens of thousands of messages per second, or hundreds of megabytes per second, Kafka is the safer choice.
6. Is per-message latency the main concern at moderate load? RabbitMQ usually delivers single messages faster.
7. What does your team already operate? A system your team runs well is often better than a better fit that nobody knows.

Three worked examples

An e-commerce order pipeline. When an order is placed, the inventory, billing, shipping, email, and analytics services all need to know. A new recommendation service may be added next quarter and will need past orders. Choose Kafka, with an orders topic keyed by order ID so that all events for one order stay in sequence. Each service is its own consumer group, and the new service replays the retained history when it starts.

A video processing service. Users upload videos, and each upload needs transcoding into four resolutions. Jobs take between 20 seconds and 10 minutes, paid users should go first, and a failed job should move to a review queue after three attempts. Choose RabbitMQ: competing consumers spread uneven jobs well, prefetch of 1 stops a worker from holding several long jobs, priorities handle paid users, and a dead-letter exchange catches repeated failures.

A ride-sharing platform. Driver location updates arrive at an illustrative 200,000 messages per second, and matching, pricing, fraud, and ETA services all read them. Choose Kafka for this stream, keyed by driver ID. The same platform sends receipts and payout notifications, which are low-volume tasks that need retries and routing by country. Many teams run RabbitMQ for those tasks next to Kafka, because each tool handles one part of the system well.

The third example shows a common outcome. Large systems often run both, with Kafka carrying the shared event streams and RabbitMQ handling task queues. Choosing one does not prevent adding the other later.

Kafka vs RabbitMQ compared

Frequently asked questions

Is Kafka better than RabbitMQ? Neither is better in general. Kafka is better for high-volume event streams that many services read and that may need replay. RabbitMQ is better for task queues that need routing, priorities, and fast delivery of individual messages.

Can RabbitMQ replace Kafka? For moderate volumes, RabbitMQ streams provide a replayable log, so RabbitMQ can cover some Kafka use cases. It does not match Kafka's throughput at large scale or its ecosystem of connectors and stream processing tools.

Can Kafka be used as a queue? Yes, with limits. A single consumer group already distributes partitions among consumers. Share groups from KIP-932 add per-record acknowledgment and let more consumers than partitions read together, which brings Kafka closer to a traditional queue.

Why is Kafka faster than RabbitMQ? Kafka writes and reads in large sequential batches and does little work per message. RabbitMQ routes, tracks, and acknowledges each message individually. That extra work gives RabbitMQ more features per message but lowers its maximum throughput.

Does Kafka guarantee message order? Kafka guarantees order within a partition. Messages with the same key go to the same partition, so they stay in order. Kafka does not guarantee order across partitions in a topic.

Should I use Kafka or RabbitMQ for microservices? Use Kafka when services publish events that several other services consume, especially when new consumers appear over time. Use RabbitMQ when one service sends commands or jobs to another and needs retries, routing, or priorities.

Which should I name in a system design interview? Name the one that matches the communication pattern in your design and say why in one sentence. For an event feed read by many services, say Kafka and mention keyed partitions. For background jobs, say RabbitMQ and mention competing consumers and dead-letter queues.

Related reading

• RabbitMQ vs Kafka vs ActiveMQ: A Complete Guide to Choosing the Right Message Broker
• Event-Driven Architecture: Patterns, Diagrams, and When to Use It Over Request-Response
• Kafka Streams vs Apache Flink vs Apache Storm: A Complete Guide to Stream Processing
• Designing a Real-Time Event Pipeline with Kafka, Flink, and Elasticsearch
• Grokking Messaging Patterns: Queues, Pub/Sub, and Event Streams

Ready to make broker choices you can defend? Grokking Kafka and Event-Driven Architecture teaches topics, partitions, delivery guarantees, and event-driven patterns, and ends with full designs you can compare against your own.