0% completed
Load Balancing
On This Page
What a Load Balancer Does
Health Checks
Where You Can Put a Load Balancer
Layer 4 and Layer 7
Keep the Servers Stateless
The Load Balancer Must Not Be the Weak Point
Hardware, Software, and Cloud
How the Server Is Chosen
Key Takeaways
One server can only handle a limited number of requests. When you need to serve more users, you add more servers. But now something has to decide which server answers the next request.
That is the job of a load balancer.
A load balancer sits between the users and your servers. Every request arrives at the load balancer first. The load balancer picks one server and sends the request there.
The users do not know that many servers exist. They only know one address, the address of the load balancer.
What a Load Balancer Does
A load balancer has three jobs.
It spreads the requests. No single server gets all the traffic, so no single server becomes slow while the others are idle.
It watches the servers. It keeps checking which servers are healthy, and it stops sending requests to a server that is not answering.
It hides the servers. Clients only know the address of the load balancer. So you can add a server, remove one, or restart one, and the client does not notice.
The third job is easy to miss. It is the reason you can grow the system without changing anything on the client side.
Health Checks
A load balancer needs to know which servers are working. It finds out by asking them, again and again.
Every few seconds, the load balancer sends a small request to each server. This is called a health check. It is usually a normal HTTP request to a path such as /health. A healthy server answers quickly with a success code.
A server may not answer, or may answer with an error. The load balancer then marks it as unhealthy and stops sending user requests to it. It keeps running the health checks. When the server answers correctly again, the load balancer starts using it again.
One detail is worth knowing. A server is usually removed only after it fails several checks in a row. One missed answer can be a slow network or a busy moment, not a broken server. Removing a server too quickly is its own problem, because the remaining servers then receive more traffic than planned.
Where You Can Put a Load Balancer
Most people think of the load balancer as the box in front of the website. You can use one at every layer where one group of machines calls another group.
There are three common places:
- Between the users and the web servers.
- Between the web servers and the application servers or cache servers.
- Between the application servers and the database.
The third one needs care. You cannot send any database request to any database machine. In most systems, writes go to one machine, and reads can be spread across the copies of that machine. Redundancy and Replication explains how this works.
Layer 4 and Layer 7
Load balancers come in two kinds. The difference is how much of the request they read.
A layer 4 load balancer looks only at the network information, which is the IP address and the port. It does not read the content of the request. It is very fast, and it can carry any kind of traffic, not only web traffic.
A layer 7 load balancer reads the request itself: the path in the URL, the headers, and the cookies. Because it understands the request, it can make better decisions. It can send /images to one group of servers and /api to another group.
Layer 7 does more work for each request, so it is slower than layer 4. For most web systems it is still the right choice. The extra work is small, and the routing is much more useful.
Keep the Servers Stateless
Load balancing works best when any server can answer any request.
This becomes a problem when a server keeps user data in its own memory. For example, the server stores the details of the user session after login. If that data lives on server 1, every later request from that user must also go to server 1. This is called a sticky session.
Sticky sessions work, but they cost you two things. If server 1 stops, that user loses the session. And a new server receives no traffic from the users who are already attached to the older servers.
The better answer is to keep session data outside the servers, for example in a shared cache. Then every server can answer every request, and the load balancer is free to choose any of them.
The Load Balancer Must Not Be the Weak Point
There is one obvious risk. If you have a single load balancer and it fails, your whole system is unreachable. Every server behind it can be healthy, and it does not help.
The usual answer is to run two load balancers.
One of them is active. It receives all the traffic. The other one is passive. It does no work, and it keeps checking whether the active one is still alive. If the active one stops answering, the passive one takes over the same address, and traffic keeps flowing.
This part confuses many people, so it is worth saying clearly. The client never chooses between the two load balancers. The client always sends the request to one address. The pair decides which machine is behind that address at this moment. The change usually takes a few seconds, and some requests fail during those seconds.
Hardware, Software, and Cloud
You can buy a load balancer as a physical device. These are fast and expensive, and you will mostly meet them in older data centers.
You can also run a load balancer as software on a normal machine. NGINX and HAProxy are the common examples.
Most teams today use a load balancer from their cloud provider, such as AWS Elastic Load Balancing. You do not run the machine, and the provider grows it when your traffic grows. In an interview, it is a normal answer to say you would use the cloud load balancer. Nobody expects you to build one.
How the Server Is Chosen
The load balancer still needs a rule for picking the server. The rule is called the load balancing algorithm. The simplest one is round robin. Send the first request to server 1, the second to server 2, the third to server 3, then start again.
There are several other rules, and they are the subject of the next lesson, Load Balancing Algorithms.
Key Takeaways
- A load balancer sits between the users and the servers. It receives every request and sends it to one of the servers.
- It has three jobs: spread the requests, check which servers are healthy, and hide the servers from the client.
- A health check is a small request that the load balancer sends to each server every few seconds. A server is removed after it fails several checks in a row. It is added back when it answers correctly again.
- You can use one at every layer. In front of the web servers, in front of the application servers, and in front of the database copies.
- A layer 4 load balancer reads only the IP address and port. A layer 7 load balancer reads the path, headers, and cookies. It can route on the content of the request.
- Load balancing works best when servers are stateless. Keep session data in a shared cache instead of in the memory of one server.
- One load balancer is a single point of failure. Run two, one active and one passive, sharing the same address.
A load balancer is one of the few pieces that appears in almost every system design answer. Draw it, say what it checks, and say what happens when it fails. Those three sentences are usually all the interviewer needs from you.
Rahul baidya
· 7 months ago
how it's being decided which Load balancer to use? If there are two
Meenu Ahalawat
· 10 months ago
I am finding content to read but no video based learning material which was mentioned on your webpage for Grokking the system design interview course.
Reading Progress
0%
On This Page
What a Load Balancer Does
Health Checks
Where You Can Put a Load Balancer
Layer 4 and Layer 7
Keep the Servers Stateless
The Load Balancer Must Not Be the Weak Point
Hardware, Software, and Cloud
How the Server Is Chosen
Key Takeaways