Load Balancer vs. API Gateway

The Load Balancer

A load balancer distributes incoming traffic across multiple backend servers, so that no single server becomes overloaded.

The servers behind a load balancer are usually identical copies. They run the same code, so any copy can handle any request. The only decision is which copy gets each request. An algorithm makes that choice. Common algorithms include:

• Round robin: send requests to each server in turn. It is simple, but it does not consider the current load or capacity of each server.
• Least connections: send each request to the server with the fewest active connections. It works well when request durations vary a lot.
• IP hash: use the client's IP address to choose the server, so the same client always reaches the same server. This gives session persistence.

The Load Balancing lesson covers these algorithms in detail.

Health Checks

A load balancer has a second job that is just as important. It sends each server a health check, which is a small request sent at regular intervals to confirm that the server still responds. When a server fails its health checks, the load balancer stops sending traffic to it.

For example, suppose the load balancer checks each server every 5 seconds, and removes a server after 3 failed checks in a row. If a server crashes, it is removed within about 10 to 15 seconds. After that, all requests go to the healthy copies.

This is what allows one server to fail without users noticing.

What a Load Balancer Reads

A Layer 4 load balancer works with network connections. It looks only at IP addresses and ports, and it never reads the request itself.

A Layer 7 load balancer reads HTTP requests. It can route by path or header, for example sending /images to one group of servers. This overlap is one reason people confuse the two boxes. But even a Layer 7 load balancer does not usually check who the user is or how many requests they have sent.

The API Gateway

An API gateway is the single entry point for your APIs. It sits in front of different services, and it reads each request to decide which service should handle it. It gives clients one unified interface, instead of many service addresses.

Because the gateway already reads every request, it is the natural place for cross-cutting concerns. These are jobs that every service needs, like security and rate limiting. Handling them once at the gateway means each service does not repeat the same code.

• Routing. Send /orders to the orders service, and /users to the users service.
• Authentication. Check who the caller is once, at the entry point, so the services can trust the request.
• Rate limiting. Reject a client that sends too many requests, before any expensive work runs.
• Caching. Return common responses without calling any service.
• Combining responses. Call several services and return one response to the client. For example, a home screen that needs data from three services can get it in one request.

A request that fails a check is stopped at the gateway. It never reaches a service, so the services do less work.

Why They Are Easy to Confuse

The two boxes overlap in some features:

• A Layer 7 load balancer can route by path, like a gateway.
• An API gateway often spreads traffic across copies of a service, as a load balancer does.
• Some products can do both jobs.

So think about responsibilities, not products. Picking a healthy copy is the load balancer's job. Checking, limiting, and routing API requests to the right service is the gateway's job. Even when one product does both, these are still two separate jobs.

Using Both, in Order

Real systems usually use both, in a standard order. A load balancer sits in front of several API gateway instances.

The reason is capacity and failure. Every request passes through the gateway. One gateway instance would be a single point of failure, which is a part whose failure stops the whole system. So the team runs several gateway instances, and the load balancer spreads traffic across them. If one gateway instance fails its health checks, the load balancer stops sending traffic to it.

Suppose one gateway instance can handle 4,000 requests per second, and peak traffic is 10,000 requests per second. The team needs 10,000 / 4,000 = 2.5, so 3 instances. To survive one failure, the team runs 4.

Behind the gateway, balancing happens again. Each service runs several copies, and requests to that service are spread across its copies. On a container platform, like Kubernetes, the platform usually provides this second layer of balancing.

Load Balancer vs. API Gateway

Choosing

• You run more than one copy of anything: add a load balancer.
• Several services sit behind one public API: add an API gateway, so clients do not need to know about each service.
• Several services need the same rules, like authentication and rate limits: add an API gateway, and write those rules once.
• You have exactly one service: skip the gateway for now. It would add cost and an extra network step, and give no new benefit.

Whether a gateway is worth adding at all is its own decision. The API Gateway vs Direct Service Exposure lesson covers it.

Using This in an Interview

When you draw both boxes, describe each one in a short phrase. For example: "A load balancer spreads traffic across the gateway instances. The gateway handles authentication, rate limits, and routing."

Two follow-up questions are common:

• "What happens when a gateway instance fails?" It fails its health checks, and the load balancer stops sending traffic to it. The other instances handle the requests.
• "Why not use one box for both?" One product can do both jobs. But the jobs are still different: picking a healthy copy, and applying API rules.

If your design has only one service, say that you are leaving the gateway out, and say what would make you add one.

Key Takeaways

• A load balancer distributes incoming traffic across multiple backend servers, so no server is overloaded. It also runs health checks and removes unhealthy servers.
• An API gateway is the single entry point for different services. It routes each request, and handles shared concerns like authentication and rate limiting.
• Both are reverse proxies. A load balancer picks a copy. A gateway picks a service and applies rules.
• A Layer 7 load balancer can also route by path, so focus on responsibilities, not products.
• Real systems usually put a load balancer in front of several gateway instances, and balance traffic again behind the gateway.
• Add a load balancer whenever you run copies. Add a gateway when several services share one public API or the same rules.

The load balancer sends requests only to healthy servers, and the gateway decides what each request is allowed to do. The next lesson, API Gateway vs Direct Service Exposure, looks at when a gateway is worth adding at all.

Practice Questions

Try each question first, then open the answer.

1. A team runs three identical copies of one web service. There is no need for authentication at the entry point, and there is only one service. Does the team need a load balancer, an API gateway, or both?

<details> <summary>Show answer</summary>

A load balancer. The three copies run the same code, so the only decision is which copy handles each request. A load balancer makes that choice and removes unhealthy copies. With only one service and no shared rules, a gateway would add cost without a clear benefit.

</details>

2. A load balancer uses round robin. Most requests take 50 ms, but some reports take 10 seconds. One server keeps getting several long reports at once and becomes overloaded. Which algorithm would help?

<details> <summary>Show answer</summary>

Least connections. Round robin sends requests in turn and ignores how busy each server is. Least connections sends each new request to the server with the fewest active connections. A server that is busy with long reports has more connections, so it receives fewer new requests.

</details>

3. A load balancer checks each server every 5 seconds, and removes a server after 3 failed checks in a row. A server crashes. About how long can the load balancer keep sending requests to it?

<details> <summary>Show answer</summary>

About 10 to 15 seconds. The load balancer needs 3 failed checks, and they are 5 seconds apart. If the server crashes just before a check, the checks fail at about 0, 5, and 10 seconds. If it crashes just after a check, they fail at about 5, 10, and 15 seconds. Requests sent in that time can fail, so clients should retry safely.

</details>

4. One API gateway instance can handle 4,000 requests per second. Peak traffic is 10,000 requests per second. How many instances should the team run, if the system must survive one failed instance?

<details> <summary>Show answer</summary>

4 instances. Peak traffic needs 10,000 / 4,000 = 2.5 instances, which rounds up to 3. If one of the 3 fails, the other 2 can handle only 8,000 requests per second. So the team adds one more instance, for a total of 4.

</details>

5. A mobile app calls three services: users, orders, and payments. Each service currently checks login tokens with its own copy of the code. What should the team add, and what does it improve?

<details> <summary>Show answer</summary>

An API gateway, with a load balancer in front of its instances. The gateway gives the app one entry point, and routes each path to the right service. It checks login tokens once, so the three services no longer repeat that code. Rate limits can also be added there. The load balancer keeps the gateway from becoming a single point of failure.

</details>