Grokking Messaging Patterns: Queues, Pub/Sub, and Event Streams
This blog explores three fundamental messaging patterns—publish-subscribe (pub-sub), message queues, and event streams—explaining what they are, how they differ, and when to use each.
Imagine clicking “Place Order” on a shopping app.
Instantly, multiple things start happening—payment processing, inventory updating, sending you a confirmation email—all handled by different services.
If each service waited on another via direct API calls, one slow service could delay or crash the whole experience.
What if the payment service is down or the email server is swamped?
Does your entire app freeze?
This is where asynchronous communication comes to the rescue.
Messaging systems let one part of your application send a message and move on without waiting for others to respond.
In simple terms, messaging decouples the sender from the receiver: one service hands off a message to a message broker and doesn’t stall – the broker safely holds the message until the receiving service is ready to process it.
If the receiver is slow or temporarily offline, the message patiently waits in line instead of breaking the whole system.
This decoupling is a game-changer for building resilient, scalable applications.
However, not all messaging systems work the same way.
In fact, there are three common messaging patterns you’ll encounter in system design, each with unique behavior:
-
Message Queues: one producer, one consumer – each message goes to a single recipient (point-to-point communication).
-
Publish-Subscribe (Pub-Sub): one producer, many consumers – messages are broadcast to all subscribers who are interested.
-
Event Streams: a continuous, durable log of events – multiple consumers can read the full history of messages at their own pace.
Each pattern shines in different scenarios and comes with trade-offs in delivery guarantees, ordering, and complexity.
Let’s explore each and see how they compare.
Message Queues: One-to-One Workhorses
A message queue is the simplest messaging pattern – think of it like a line at a coffee shop.
Orders (messages) form a queue, and a barista (consumer) takes each order one by one.
If more baristas are hired, they all pull from the same queue to serve customers faster.
In tech terms, one service (producer) sends messages into the queue, and one consumer service receives each message to process it, ensuring each task is handled once and only once.
Once a message is consumed and processed, it’s removed from the queue.
This pattern is fantastic for work distribution and load leveling.
For example, suppose an e-commerce site needs to generate PDFs of invoices. The order service can post “generate invoice” jobs into a queue, and multiple invoice-worker services will each grab jobs from the queue to process in parallel.
This way, if there’s a spike in orders, the work is automatically load-balanced among workers.
Queues excel at smoothing out traffic bursts and preventing any single worker from getting overwhelmed.
They also typically provide strong delivery guarantees – the system makes sure that a message isn’t lost and eventually gets processed even if workers crash or go offline temporarily.
Technologies:
Many systems implement queues – for instance, RabbitMQ and Apache ActiveMQ are popular message brokers for queues, and cloud services like Amazon SQS provide a fully-managed queue service.
(For a deeper dive into message queues and their role in system design, see the course Grokking System Design Fundamentals.)
Publish-Subscribe (Pub-Sub): One-to-Many Broadcasting
The publish-subscribe pattern, often called pub-sub, works more like a TV broadcast or a YouTube live stream.
Here, a producer (publisher) sends each message out not to a specific receiver but to a topic or channel that multiple listeners have subscribed to.
Every subscriber interested in that topic gets their own copy of the message.
In other words, pub-sub is a one-to-many messaging model.
The publisher doesn’t need to know who is listening; it just publishes events, and the messaging system (broker) delivers those events to all current subscribers.
This pattern shines when you want to broadcast information to multiple services at once without hard-coding who they are.
For example, imagine an online store where a new order event needs to notify several systems: the inventory service should decrement stock, the shipping service should schedule delivery, and the analytics service should log the sale.
Using pub-sub, the order service can publish an “Order Placed” event to the Orders topic, and all three subscriber services will instantly receive it and act accordingly.
The publisher simply says “Hey, an order was placed!” and any service that cares about orders is kept in the loop, all asynchronously.
The big benefit of pub-sub is decoupling and flexibility.
New subscribers can join or leave without affecting the publisher or requiring code changes in the publishing service. It’s great for building scalable, event-driven architectures where components react to events in real time.
However, pub-sub usually offers weaker delivery guarantees compared to queues – if no subscriber is listening at the moment an event is published, that event might be lost (unless you use a system that can buffer or store events).
Also, each message is typically ephemeral in pure pub-sub; once delivered to current subscribers, it’s not stored for future subscribers.
Technologies:
Pub-sub messaging is implemented by systems like Redis Pub/Sub (for simple low-latency messaging), Google Cloud Pub/Sub, or AWS SNS (Simple Notification Service).
Even Apache Kafka can act as a pub-sub broker, delivering messages to multiple consumers, though Kafka is more famously used for event streaming (coming up next).
Event Streams: The Durable, Replayable Log
Event streams take the pub-sub idea one step further by not only allowing multiple consumers but also storing the events in a durable log.
You can think of an event stream like a never-ending tape recording or an ever-growing ledger of events.
When a producer writes an event to the stream, it gets appended to the end of the log.
Subscribers can listen in real-time and also rewind to any point in the log to replay past events.
Unlike a transient pub-sub message that’s gone once delivered, an event stream retains all events for a defined time or indefinitely.
This means consumers aren’t required to be online at the exact moment an event is published – they can come later, read from the backlog, and even reprocess events as needed.
This pattern is powerful for scenarios like analytics, auditing, and state reconstruction.
For instance, consider a bank that keeps an event stream of all account transactions. Each deposit or withdrawal is an event in the stream. A new service (say, a fraud detection service) can start consuming from the beginning of the transaction stream and analyze every historical event in order.
Technologies:
The poster child of event streaming is Apache Kafka, a distributed platform designed for high-volume, durable event streams.
Kafka stores events in topics and lets consumers rewind and replay, with strong ordering guarantees per topic-partition.
Other technologies include Amazon Kinesis and Apache Pulsar, which serve similar purposes in cloud and large-scale environments.
When to Use What: Choosing the Right Pattern
So, how do you decide which messaging pattern to use? It often boils down to your use case requirements:
-
Use a Message Queue if you need reliable, one-time processing of each message by a single consumer. Queues are perfect for task distribution – e.g. processing jobs from a work queue – especially when you require strong delivery guarantees (no lost tasks) and automatic load balancing across workers. If you have many identical consumers and want to spread work among them (like multiple email senders or image processors), a queue is usually the way to go.
-
Use Pub-Sub when you need to broadcast events to multiple receivers at once, such as notifications or system events that various services might need to know about simultaneously. It’s great for real-time updates and decoupling – your producers and consumers remain unaware of each other, making it easy to add new consumers.
-
Use Event Streams when you need a durable sequence of events that can be replayed or consumed by different services at different times. This is ideal for event-driven architectures that require audit logs, playback, or state rebuilding. For example, choose event streams for logging user activities, tracking sensor data, or implementing event sourcing in a financial system.
Many real-world systems actually combine these patterns to get the best of each.
You might use a queue for one-to-one communication (e.g. a service commanding a background job), but use pub-sub to notify multiple services of important events, and use an event stream to maintain a long-term record of critical events or feed analytics pipelines.
The key is understanding the trade-offs: delivery guarantees, persistence, number of consumers, and processing model.
Final Thoughts
Messaging patterns are the backbone of scalable and fault-tolerant system design.
Mastering queues, pub-sub, and event streams will make you a better architect, because you’ll know how to keep services loosely coupled yet coordinated.
Always remember that there’s no one-size-fits-all – the best pattern depends on what your system needs to achieve in terms of data flow, timing, and reliability.
If you’re excited to learn more and deepen your expertise, consider exploring courses like Grokking the System Design Interview or Grokking the Advanced System Design Interview by DesignGurus.io.
What our users say
Brandon Lyons
The famous "grokking the system design interview course" on http://designgurus.io is amazing. I used this for my MSFT interviews and I was told I nailed it.
Arijeet
Just completed the “Grokking the system design interview”. It's amazing and super informative. Have come across very few courses that are as good as this!
Eric
I've completed my first pass of "grokking the System Design Interview" and I can say this was an excellent use of money and time. I've grown as a developer and now know the secrets of how to build these really giant internet systems.
