0% completed
Introduction to Messaging System
On This Page
The Problem: Services Talking Directly
What Is a Messaging System?
The Queue Model
The Publish-Subscribe Model
The Message Broker
Why Systems Use a Messaging System
A large application is rarely a single program. It is a collection of services that must constantly pass data to each other. The part of the system that carries this data between services is called a messaging system. This lesson explains what a messaging system is, the two main models it uses, and why almost every large system depends on one.
The Problem: Services Talking Directly
Imagine a log aggregation service. It receives hundreds of log entries per second from many different sources. Its job is to store these logs on disk on a shared server and build an index so the logs can be searched later.
Connecting the sources directly to this service creates three hard problems:
- Spikes. Suppose the service can handle (or buffer) 500 messages per second. What happens when the sources suddenly send more than that? And if we run multiple instances of the service to keep up, how do we divide the work among them?
- Different senders. Every source must agree with the service on a common protocol and data format for sending log messages. This creates a strongly coupled architecture between the producer and the consumer of the messages: change one side, and you must change the other. (Coupling means how tightly two components depend on each other.)
- Downtime. What happens to the log messages if the aggregation service is down or unresponsive for a while? Without a safety net, they are simply lost.
To manage such scenarios efficiently, distributed systems depend on a messaging system.
What Is a Messaging System?
A messaging system is a service that transfers data among services, applications, processes, or servers. It sits between senders and receivers and takes over the delivery work. Senders (also called producers) and receivers (also called consumers) focus only on the message itself, not on the mechanics of moving it.
This gives the system two big properties:
- Decoupling. Producers and consumers do not need to know about each other. Each side can change, move, or scale without breaking the other.
- Asynchronous delivery. The producer hands the message over and moves on. It does not wait for the consumer to be ready. (Asynchronous means the sender does not block while waiting for the receiver.)
There are two common ways to handle messages: queuing and publish-subscribe.
The Queue Model
In the queuing model, messages are stored in a queue, one after another, in the order they arrive. Producers push messages to the rear of the queue, and consumers extract messages from the front. It works like a checkout line in a store: first in, first served.
A message in a queue is consumed by at most one consumer. Once a consumer grabs a message, the message is removed from the queue, and the next consumer gets the next message. This makes queues a great fit for spreading message processing across multiple consumers. It also sets the model's limit: multiple consumers can never read the same message from a queue.
The Publish-Subscribe Model
In the pub-sub model (short for publish-subscribe), messages are divided into named categories called topics. A publisher (the producer) sends a message to a topic, and the messaging system stores it under that topic. Subscribers (the consumers) subscribe to a topic and receive every message published to it.
Unlike the queue model, pub-sub allows multiple consumers to get the same message. If two consumers subscribe to the same topic, both receive every message published to that topic. It works like subscribing to a newsletter: every subscriber gets their own copy of every issue.
The Message Broker
The component that stores and maintains the messages is called the message broker. The broker is what makes the loose coupling possible. Publishers write messages to the broker, subscribers read messages from the broker, and the two sides never talk directly.
Because the broker stores messages, publishers and subscribers do not have to be synchronized. A fast producer can keep publishing while a slow consumer catches up, so both sides read and write at their own rates. And because messages are stored, they do not get lost between the time they are produced and the time they are consumed. This ability to hold messages safely is what gives a messaging system its fault tolerance.
Why Systems Use a Messaging System
To summarize, a messaging system is deployed in an application stack for the following reasons:
- Message buffering. The broker acts as a buffer in front of processing. When incoming messages spike beyond what the processing app can handle, the extra messages wait safely in the broker until the app is ready for them.
- Guaranteed delivery. A producer can publish with confidence even when the consuming application cannot receive the message right now. The message waits in the broker and is delivered eventually.
- Abstraction. Senders and receivers are decoupled, so each component can evolve independently. This keeps the system modular and makes it easier to maintain and update one component without affecting the rest.
- Scalability. A distributed messaging system can handle a very large number of messages and can scale horizontally (by adding more machines) as the workload grows, without significant performance loss.
- Fault tolerance. Messages are distributed across multiple nodes or servers. If a single node fails, the system keeps operating, and messages are not lost.
- Asynchronous communication. Components process messages at their own pace instead of waiting for immediate responses from each other. This improves overall performance and responsiveness, especially when processing times vary or latency is high.
- Load balancing. The system automatically distributes messages across multiple nodes and consumers, so no single node becomes a bottleneck and resources are used well.
- Message persistence. Messages are stored durably, so nothing is lost when a receiver is temporarily unavailable or slow to process. This helps keep data consistent and reliable across the system.
- Security. Messaging systems commonly support mechanisms such as encryption and authentication to protect sensitive data and prevent unauthorized access.
- Interoperability. Most messaging systems support multiple messaging protocols and integrate with many platforms and technologies, which makes it easier to connect the different components of a complex system.
In the next lesson, we will study the most widely used messaging system in modern infrastructure: Introduction to Kafka. Later in this chapter, Messaging patterns shows the common ways these building blocks are combined.
Sid D
· a year ago
Do messaging systems ensure a chronological order when various producers try to enqueue a message?
Is ordering guaranteed for the consumer?
Sid D
· a year ago
In a pub-sub message system, when is a message dequeued since there might be another consumer waiting to read a message?
Valentino Christian
· 2 years ago
I understand how nodes can be horizontally scaled, and how a messaging system could be vertically scaled to allow more throughput through a particular topic, but how the queue or pub-sub messaging system itself be horizontally scaled?
Anand Mohan
· 3 years ago
The messaging system that stores and maintains the messages is commonly known as the message broker. It provides a loose coupling between publishers and subscribers, or producers and consumers of data.
Reading Progress
0%
On This Page
The Problem: Services Talking Directly
What Is a Messaging System?
The Queue Model
The Publish-Subscribe Model
The Message Broker
Why Systems Use a Messaging System