On this page
What Happens Behind the Scenes
Static Load Balancing Algorithms
The Round Robin Algorithm
The Weighted Round Robin Algorithm
The IP Hash Algorithm
Dynamic Load Balancing Algorithms
The Least Connections Algorithm
The Weighted Least Connections Algorithm
The Least Response Time Algorithm
Active Server Health Checks
Conclusion
System Design Fundamentals: The Load Balancing Algorithms Guide
On This Page
What Happens Behind the Scenes
Static Load Balancing Algorithms
The Round Robin Algorithm
The Weighted Round Robin Algorithm
The IP Hash Algorithm
Dynamic Load Balancing Algorithms
The Least Connections Algorithm
The Weighted Least Connections Algorithm
The Least Response Time Algorithm
Active Server Health Checks
Conclusion
A fundamental problem in software architecture occurs when network traffic exceeds physical hardware limits. Every computer server possesses a strict maximum capacity for processor utilization and system memory.
When an application receives millions of simultaneous network requests, a single server simply cannot process the computational load.
The processor usage spikes to its absolute limit, and the memory fills up completely.
Once hardware resources become completely exhausted, the system fails under the heavy network load. This hardware failure results in dropped network connections and a completely broken user interface.
Software engineers solve this hardware limitation by adding multiple servers to share the total workload.
This specific architectural approach is called horizontal scaling.
Adding more servers introduces a new technical challenge for the system architecture.
The network needs a highly reliable way to distribute massive volumes of incoming requests evenly across all available machines. If one server receives all the traffic while the others sit idle, the system will still crash.
A load balancer is a dedicated software component that sits in front of the backend servers. It acts as the single point of entry for all incoming network traffic.
What Happens Behind the Scenes
A load balancer receives every single client request and decides which backend server should process it.
To make this routing decision quickly and efficiently, the load balancer relies on specific mathematical rules. These sets of rules are called load balancing algorithms.
Before looking at specific algorithms, it is important to understand how a load balancer manages network traffic.
The load balancer maintains a list of all available backend servers in its memory. This list of active machines is often called a server pool or a cluster.
The load balancer constantly monitors these servers to ensure they are online and ready to accept traffic. It achieves this monitoring by sending regular automated network requests called health checks.
A health check is a tiny packet of data sent to verify server availability. If a server fails to respond to a health check, the load balancer marks that machine as unhealthy. The routing software immediately stops sending new user requests to the broken server.
Once the server recovers and passes the health check, it is added back into the active rotation.
When a client sends a request to an application, the network packet hits the load balancer first.
The load balancer reads the request data and inspects its own internal configuration settings. It then runs the chosen load balancing algorithm to select the best destination server.
The load balancer forwards the client request to that specific server over an internal network connection.
The backend server processes the data and sends the computed response back to the load balancer.
Finally, the load balancer returns that computed response to the original client device. Engineers divide load balancing algorithms into two primary categories known as static and dynamic algorithms.
Static Load Balancing Algorithms
Static load balancing algorithms distribute network traffic based on fixed mathematical rules. They do not monitor the current state, processor load, or active connections of the backend servers. They simply follow a rigid mathematical pattern to route traffic regardless of real time server health. These algorithms are generally easier to implement and require very little computational overhead.
The Round Robin Algorithm
The Round Robin algorithm is the simplest and most common static routing method in system design. It distributes incoming requests sequentially across the server pool in a continuous loop.
The load balancer maintains a simple integer counter in its memory to track the routing order. When the first network request arrives, the load balancer sends it to the first server in the list.
The second incoming request goes to the second server. The third incoming request goes to the third server. Once the load balancer reaches the end of the server list, it loops back to the very beginning. It resets the internal counter and sends the next request to the first server again.
This algorithm works exceptionally well when all backend servers have identical hardware specifications. It assumes that every backend server can handle the exact same amount of network traffic. It also assumes that every incoming request takes the exact same amount of time to process. However, this algorithm has a major flaw in highly complex software systems.
If one request requires heavy database processing and another requires a simple text response, Round Robin treats them equally.
A single server might receive several heavy database requests in a row by pure chance. That specific server will become severely overloaded while other servers in the cluster remain completely idle.
The Weighted Round Robin Algorithm
The Weighted Round Robin algorithm is an advanced variation of the standard sequential approach. It solves the problem of unequal hardware within the backend server pool.
In a real software system, engineers often mix older servers with newer and more powerful machines.
The standard sequential method would overwhelm the older machines by giving them equal workloads.
To fix this, system administrators manually assign a numerical weight to each server in the configuration file. This weight integer represents the total processing capacity of the specific machine.
For instance, a new server with high memory capacity might receive a mathematical weight of three. An older server with limited processing capacity might receive a mathematical weight of one.
The load balancer reads these integer weights to determine the exact routing sequence.
In this specific scenario, the load balancer will send three sequential requests to the powerful server. It will then send exactly one request to the older server before looping back to the beginning.
This mathematical adjustment ensures that powerful servers handle a much larger percentage of the total traffic load.
While this solves the hardware disparity problem, it remains a purely static algorithm. It still does not know if a powerful server is currently struggling with a heavy computational task. It blindly follows the weighted pattern regardless of real time system performance.
The IP Hash Algorithm
The IP Hash algorithm routes network traffic based on the network address of the incoming client. Every device connected to the internet has a unique numerical identifier called an IP address.
The load balancer reads this exact IP address from the incoming network packet header.
The load balancer then passes this IP address through a mathematical function called a hash function.
A hash function takes a variable input and converts it into a fixed numerical value.
The load balancer takes this resulting number and applies a mathematical modulo operation against the total number of servers.
The modulo operation divides the hash number by the server count and returns the exact remainder.
If a system has four servers, the remainder will always be zero, one, two, or three.
This final remainder mathematically dictates which server receives the incoming request. This mathematical consistency guarantees that a specific client IP will always route to the exact same backend server. This concept is highly useful for maintaining session persistence in a software application.
If a server stores specific login data in its local memory, the client must keep communicating with that exact machine. If a different algorithm routes the client to a different server, the client would lose their active session state.
Dynamic Load Balancing Algorithms
Dynamic load balancing algorithms are much smarter and more complex than static algorithms. They constantly monitor the real time health, active load, and performance metrics of every server in the pool.
The load balancer uses this live data to make highly optimized and dynamic routing decisions. These algorithms prevent individual servers from becoming overwhelmed by long running background processes.
The Least Connections Algorithm
The Least Connections algorithm monitors the exact number of active connections on every single backend server.
When a client request arrives, it opens a secure network connection with the destination server.
This connection remains completely open while the server computes the necessary response. Some network requests process in milliseconds, while others take several seconds to complete.
The load balancer maintains a dynamic database in its active memory. This database maps each server identifier to an active connection counter.
When the load balancer forwards a request to a server, it increments that server connection counter by one.
When the server finishes processing and closes the connection, the load balancer decrements the counter by one.
When a brand new user request enters the system, the load balancer scans its internal database. It identifies the server with the absolute lowest number of active network connections. It then immediately routes the new request to that specific machine.
This algorithm is ideal for applications where request processing times vary wildly.
It naturally protects backend servers that are stuck processing heavy and time consuming tasks.
The load balancer will automatically route new traffic to idle servers until the busy server finishes its current workload.
The Weighted Least Connections Algorithm
The Weighted Least Connections algorithm combines hardware capacity metrics with real time connection tracking.
The standard least connections method assumes all servers have identical hardware capabilities.
A low power server might become critically overwhelmed with ten active connections.
A high power server could easily manage one hundred active connections without slowing down.
To implement this advanced logic, engineers assign numerical capacity weights to the backend servers.
When a new network request arrives, the load balancer performs a rapid mathematical calculation. It takes the current number of active connections on a server and divides it by that server weight. It performs this exact calculation for every single machine in the cluster.
The load balancer then routes the incoming traffic to the server with the lowest calculated ratio.
This intelligent mathematical calculation balances the real time workload perfectly against the physical hardware limits. It ensures that powerful machines continually receive more network traffic even if their total connection count is higher.
The Least Response Time Algorithm
The Least Response Time algorithm is one of the most sophisticated routing methods available in system design. It combines two different dynamic metrics to make the best possible routing decision. It looks at both the active network connections and the historical processing speed of each server.
A server might have very few active connections, but its processor might be struggling due to a memory leak.
The load balancer continuously measures exactly how long it takes for a server to return a data response. It calculates a rolling mathematical average of this response time over a specific time window.
The standard least connections algorithm would blindly send traffic to a slow server if its connection count was low. The least response time algorithm notices the slow processing speed and actively avoids that machine.
When a new request arrives, the load balancer evaluates the combined data. It selects the server that has the fewest active connections and the fastest average response time. This complex calculation ensures the client request is handled as quickly as possible.
This algorithm provides the most optimal user experience in complex and globally distributed software architectures.
However, it requires significant computing power on the load balancer itself. The load balancer must constantly calculate averages and update internal metrics for hundreds of servers every single millisecond.
Active Server Health Checks
Algorithms alone cannot guarantee high availability in software architecture.
A load balancer requires supporting mechanisms to execute these routing decisions safely. The most important supporting mechanism is the active health check system.
A load balancer must know if a server is actually alive before sending data to it.
Hardware failures are an unavoidable reality in system design. Network cables degrade, power supplies fail, and operating systems crash unexpectedly.
If a load balancer routes traffic to a completely dead server, the user request will simply fail. To prevent this, the load balancer performs continuous active health checks.
Behind the scenes, the load balancer acts as a monitoring agent. Every few seconds, it sends a specific network command to every server in the pool. This command specifically asks the server for a status update.
If the server is healthy, it replies with a standard network success code.
If the server crashes, it will fail to reply within a designated time limit. The load balancer instantly registers this timeout as a critical hardware failure. It forcefully modifies its internal memory and removes the dead server from the active routing pool.
From that moment forward, the mathematical algorithms completely ignore the dead server. All new traffic is smoothly distributed among the remaining healthy machines. Once the broken server is repaired, it begins replying to the health checks again. The load balancer detects the successful replies and automatically reinserts the machine back into the active pool.
Conclusion
-
Load balancers operate as proxy systems to distribute network traffic across multiple backend servers.
-
Routing algorithms provide the mathematical instructions required to determine the optimal server destination.
-
Static algorithms rely on fixed mathematical rules without monitoring live backend server hardware conditions.
-
Round Robin distributes traffic sequentially to ensure an even numerical distribution of network requests.
-
IP Hash mathematically ensures session persistence by calculating distinct values from source addresses.
-
Dynamic algorithms continuously monitor active server metrics to make highly intelligent routing decisions.
-
Least Connections routes incoming traffic based on open network sockets to balance active workloads.
-
Least Response Time ensures the absolute lowest latency by measuring server processing speed.
-
Health checks constantly monitor backend availability to prevent algorithms from selecting crashed hardware.
What our users say
AHMET HANIF
Whoever put this together, you folks are life savers. Thank you :)
Nathan Thomas
My newest course recommendation for all of you is to check out Grokking the System Design Interview on designgurus.io. I'm working through it this month, and I'd highly recommend it.
Arijeet
Just completed the “Grokking the system design interview”. It's amazing and super informative. Have come across very few courses that are as good as this!
Designgurus on Substack
Deep dives, systems design teardowns, and interview tactics delivered daily.
Access to 50+ courses
New content added monthly
Certificate of completion
$29.08
/month
Billed Annually
Recommended Course
Grokking the Object Oriented Design Interview
58,653+ students
3.9
Learn how to prepare for object oriented design interviews and practice common object oriented design interview questions. Master low level design interview.
View Course
