Introduction to Load Balancing

Session Persistence

Session persistence, also called sticky sessions, makes sure that later requests from the same client go to the same backend server. The load balancer usually does this with a cookie that records which server the client used.

This helps when a server keeps a user's session data, like a shopping cart, in its own memory. But it has costs. Load can become uneven, and users lose their session if their server fails. A better long-term design keeps session data in a shared store, so any server can handle any request. The stateless vs. stateful load balancing lesson explains this choice.

SSL/TLS Termination

SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are the protocols that encrypt HTTPS traffic. TLS is the modern version of SSL.

SSL/TLS termination means decrypting HTTPS traffic at the load balancer, instead of on each backend server. This removes the decryption work from the backend servers, so they have more CPU for application work. It also keeps certificates in one central place. The HTTP vs. HTTPS lesson explains how TLS works.

Where to Place Load Balancers

A system can use a load balancer at each layer, not only at the front. To get full scalability and redundancy, you can add load balancers in three places.

1. Between the users and the web servers. This is the most common place.
2. Between the web servers and the internal platform layer, like application servers or cache servers.
3. Between the internal platform layer and the database servers.

Each layer can then grow and survive a failed server on its own.

The database layer needs more care. Usually only the primary database accepts writes. So a load balancer in front of databases normally spreads read requests across read replicas, while writes still go to the primary.

Kinds of Load Balancers

Load balancers come in three main forms.

• Hardware load balancers are dedicated physical devices. They are very fast, but expensive and less flexible.
• Software load balancers run on ordinary servers. Common examples are Nginx, HAProxy, and Envoy.
• Cloud load balancers are managed services from cloud providers. The provider runs and scales them for you.

Load balancers also differ in how much of each request they read. A Layer 4 load balancer routes by IP address and port. A Layer 7 load balancer can read HTTP details, like the URL path and cookies. The load balancer types lesson covers these differences.

The Load Balancer Must Not Fail

If every request passes through one load balancer, that load balancer becomes a new single point of failure. So production systems run at least two.

A common setup is active-passive. One load balancer handles all traffic, and a second one waits as a standby. If the active one fails, the standby takes over the same virtual IP address, and clients keep using the same address. The challenges of load balancers lesson covers this and other problems.

Key Takeaways

• A load balancer is a device or software that sits between clients and backend servers, and distributes incoming traffic across the servers.
• Its main goal is to prevent any server from being overloaded or becoming a single point of failure, which improves availability, reliability, and performance.
• The backend servers are also called the server pool or server farm.
• A load balancer receives a request, chooses a server with an algorithm, forwards the request, and returns the server's response to the client.
• Health checks remove unhealthy servers from the pool until they recover.
• Session persistence sends a client's requests to the same server. SSL/TLS termination decrypts HTTPS traffic at the load balancer.
• Load balancers can sit at each layer: before the web servers, before the application layer, and before the databases.
• Run at least two load balancers, so the load balancer is not a single point of failure.

A load balancer lets a group of servers work together and serve users through one reliable address. The next lesson, Load Balancing Algorithms, explains the different ways it can choose a server for each request.

Practice Questions

Try each question first, then open the answer.

1. Three servers sit behind a load balancer, and each server can handle 1,000 requests per second. Peak traffic is 2,400 requests per second. Can the system still handle the peak if one server fails?

<details> <summary>Show answer</summary>

No. With one server down, two servers can handle 2 x 1,000 = 2,000 requests per second, which is less than 2,400. To survive one failure at peak, the pool needs at least four servers. Then the three remaining servers can handle 3,000 requests per second.

</details>

2. A load balancer runs a health check every 10 seconds, and removes a server after 3 failed checks in a row. A server crashes. About how long does it keep receiving traffic?

<details> <summary>Show answer</summary>

About 20 to 30 seconds. The first failed check happens within 10 seconds of the crash. Two more checks follow, 10 seconds apart. So the server is removed after the third failure, about 20 to 30 seconds after the crash. Passive health checks can detect the problem sooner, because they watch real requests.

</details>

3. A shopping app stores each user's cart in the memory of its server. After a load balancer is added, users say their carts keep disappearing. Which load balancer feature helps right away, and what is the better long-term fix?

<details> <summary>Show answer</summary>

Session persistence helps right away, and a shared store is the better fix. Sticky sessions send each user back to the same server, so the cart stays visible. But load can become uneven, and carts are lost when a server fails. Storing carts in a shared store, like Redis or a database, lets any server handle any request.

</details>

4. The backend servers spend a lot of CPU time decrypting HTTPS traffic. Which load balancer feature reduces this work?

<details> <summary>Show answer</summary>

SSL/TLS termination. The load balancer decrypts the HTTPS traffic itself, and forwards the requests to the backend servers. The servers no longer do the decryption, so they have more CPU for application work. Certificates are also managed in one place.

</details>

5. An app has web servers, application servers, and database replicas. Where can load balancers be placed? Why does a load balancer in front of the databases usually spread only read requests?

<details> <summary>Show answer</summary>

At three places: before the web servers, before the application servers, and before the database servers. In most database setups, only the primary accepts writes, and the replicas hold copies for reading. So the load balancer spreads read requests across the replicas, while writes still go to the primary.

</details>