Availability

The numbers show why each nine is harder to reach than the last. At 99.9 percent, a team has about 43 minutes each month to notice a problem and fix it. At 99.999 percent, it has about 26 seconds. A person cannot even open an alert in that time. So a five-nines system must find failures and recover from them without human help.

More nines also cost more. Each nine usually needs more servers, more data centers, and more engineering work. The right target depends on what the product does. A payment service needs a higher target than an internal reporting tool that people check once a week.

Planned and Unplanned Downtime

Downtime has two sources.

• Planned downtime is chosen by the team. Examples are software upgrades, database changes, and hardware replacement.
• Unplanned downtime is not chosen. Examples are a server crash, a bad code release, a full disk, a network failure, or a power cut.

Users do not care which kind it is. The app is down either way. For this reason, many systems avoid planned downtime too. They upgrade one server at a time while the other servers keep serving requests. This is called a rolling update.

Availability of Connected Parts

A real system is made of many parts. One request may pass through a load balancer, an application server, and a database. The availability of the whole system depends on how these parts are connected.

Parts in series. Parts are in series when a request needs every one of them. If any one part fails, the request fails. To get the total availability, multiply the availability of each part.

A(total) = A1 x A2 x A3

Suppose a load balancer is 99.99 percent available, an app server is 99.9 percent, and a database is 99.9 percent. The total is 0.9999 x 0.999 x 0.999, which is about 99.79 percent. That is about 18 hours of downtime a year.

Look at that result again. Every part was at least 99.9 percent available, but the whole system is below 99.9 percent. Each required part lowers the total availability.

Parts in parallel. Parts are in parallel when any one of them can handle the request. The request fails only when all of them fail at the same time. To get the total, multiply the chances of failure, then subtract the result from 1.

A(total) = 1 - (1 - A1) x (1 - A2)

Suppose one server is 99 percent available, so it is down 1 percent of the time. Two such servers are both down only 1% x 1% = 0.01% of the time. So the pair is available 99.99 percent of the time. Two ordinary servers together give four nines.

The parallel formula has one condition. The copies must fail independently, which means the failure of one copy does not cause the other to fail. Two servers in the same rack share one power supply. If that power fails, both servers stop together. Two servers running the same buggy code also crash together. So real redundancy puts copies in different places, with separate power and network paths.

Single Points of Failure

A single point of failure (SPOF) is one part whose failure stops the whole system.

Here is an example. A website has three app servers and one database. If the database fails, all three app servers have no data to read. The extra app servers did not help, because the database was a single point of failure.

To find these parts, follow the path of one request through the system. For each part on the path, ask one question: "If only this part fails, does the request still work?" Every part where the answer is "no" is a single point of failure.

How Systems Stay Available

No system can prevent every failure. A highly available system expects parts to fail, and it keeps serving users when they do. The techniques below work together.

Redundancy

Redundancy means keeping more than one copy of every important part. There are two common ways to run the copies.

• Active-active. All copies serve traffic at the same time. If one copy fails, the others take its share. Each copy needs spare capacity for this extra load.
• Active-passive. One copy, the active one, serves all traffic. Another copy, the standby, waits. If the active copy fails, the standby takes over.

Active-active uses every machine, but losing one copy puts more load on the others. Active-passive is simpler, but the standby is idle most of the time. Switching to it also takes some time.

Replication

Data needs copies too. Replication means keeping copies of the same data on more than one machine. Usually one machine, the primary, accepts writes. The other machines, the replicas, receive copies of those writes. Often a copy reaches a replica a little later, so a replica can be a few seconds behind the primary. This delay is called replication lag. If the primary fails, a replica can become the new primary. The replication lesson covers the details.

Load Balancing and Health Checks

A load balancer is a server that spreads incoming requests across many servers. It sends a small test request, called a health check, to each server every few seconds. When a server stops answering, the load balancer stops sending it traffic. Users keep getting answers from the healthy servers. The load balancing lesson explains how this works.

Monitoring and Alerts

A team cannot fix a problem it does not know about. Monitoring means collecting numbers about the system all the time, like error rate, response time, and CPU use. An alert is a message sent to an engineer when one of those numbers crosses a limit.

For example, a team can send an alert when more than 1 percent of requests fail for 5 minutes. Good alerts find problems before most users notice them. Regular maintenance, like applying security patches, also prevents some failures before they happen.

Failover

Failover is the process of moving traffic from a failed part to a healthy copy. It happens in four steps.

1. Detect. The system notices the failure. A server that misses several health checks in a row is marked as down.
2. Decide. The system picks a healthy copy to take over. For a database, this often means making a replica the new primary.
3. Switch. Traffic moves to the healthy copy. The load balancer or DNS settings are updated.
4. Restore. The team or the system repairs the failed part and adds it back as a copy.

Users see errors during the first three steps. So the downtime of one failure is the time to detect, plus the time to decide, plus the time to switch.

Here is an example. Suppose health checks run every 5 seconds, and a server is marked down after 3 missed checks. Detection alone then takes about 15 seconds. With automatic failover, the whole switch takes only seconds. A manual failover, where a person must see an alert and act, can take 30 minutes or more.

Separate Zones and Regions

Copies in the same building can fail together, for example during a fire or a power cut. So cloud providers split their data centers into zones and regions.

• An availability zone is one data center, or a group of data centers, with its own power, cooling, and network.
• A region is a geographic area, like Mumbai, that contains several zones.

Copies in two zones keep working if one data center fails. Copies in two regions keep working if a whole region fails. Spreading copies across locations like this is called geographic distribution. Multi-region setups cost more, and copying data over long distances adds delay.

Serve Less Instead of Failing

When one feature fails, the rest of the app can often keep working. Suppose the recommendation service of a shopping app is down. The product page can still load without the "You may also like" section. Users can still search, add items to the cart, and pay.

This is called graceful degradation. The main actions keep working while an optional feature is down. A page with one missing section is much better than an error page.

Load Shedding and Rate Limiting

Too much traffic is a common cause of downtime. When servers get more requests than they can handle, every request becomes slow. Soon, all of them start to fail.

Load shedding means rejecting some requests on purpose, with a quick "try again later" error. The system then has enough capacity to serve the remaining requests well. Rate limiting is a related idea. It limits how many requests one user or client can send in a time window.

Safe Releases

Many outages start with a new code release. Teams reduce this risk by releasing in small steps.

A canary release sends the new version to a small share of users first, like 5 percent. The team watches the error rate. If errors rise, the team does a rollback, which means going back to the previous version. Only a small group of users sees the problem.

MTBF and MTTR

Two more numbers help explain availability.

• MTBF (mean time between failures) is the average time a system runs before it fails.
• MTTR (mean time to repair) is the average time it takes to bring the system back after a failure.

Availability = MTBF / (MTBF + MTTR)

Suppose a system fails once every 1,000 hours on average. Each time, it takes 1 hour to bring it back. Its availability is 1,000 / 1,001, which is about 99.9 percent.

Now suppose the failures happen just as often, but automatic failover restores service in 6 minutes, which is 0.1 hours. The availability becomes 1,000 / 1,000.1, which is about 99.99 percent.

Faster recovery added a nine without reducing the number of failures. This is why monitoring and automatic failover matter so much. Failures cannot be stopped completely, but they can be made short.

SLI, SLO and SLA

Teams write availability targets down using three related terms.

• An SLI (service level indicator) is the number you measure. An example is the percentage of successful requests over the last 30 days.
• An SLO (service level objective) is the team's own target for that number. For example, 99.95 percent of requests succeed.
• An SLA (service level agreement) is a promise in a contract with customers. If the provider misses it, the provider pays a penalty, often as a discount on the bill.

The SLA is usually lower than the SLO. For example, a team may set its SLO at 99.95 percent and its SLA at 99.9 percent. If the team misses its own target, it gets a warning before it breaks the customer promise.

The downtime a target allows is called the error budget. A monthly target of 99.9 percent gives an error budget of 43.2 minutes. When most of the budget is used, the team slows down new releases and works on stability first.

Availability, Reliability and Consistency

These three words are often mixed up.

• Availability asks: does the system respond?
• Reliability asks: does the system do the right thing, without failures, over a long time? A system can be available but not reliable. For example, it may answer every request quickly but sometimes return wrong data.
• Consistency asks: do all copies of the data show the same value?

More copies improve availability, but copies can disagree for a short time. So every system with copies must decide how closely the copies must agree. This decision is called the consistency model.

• Strong consistency means every copy shows the same data at all times. A write must reach every copy before it counts, so the system is less available and slower.
• Weak consistency allows copies to disagree for a while. In return, the system is more available and faster.
• Eventual consistency allows copies to disagree for a short time, but all copies reach the same data over time. It sits between the other two.

The hardest case is a broken network between copies. Then the system must choose. It can answer with data that may be old, and stay available. Or it can refuse to answer until the copies agree, and stay consistent. The CAP theorem describes this choice.

High availability is also different from fault tolerance, where users see no interruption at all. The Fault Tolerance vs. High Availability lesson compares the two.

Key Takeaways

• Availability is the share of time, or of requests, that a system serves users successfully.
• Each extra nine allows 10 times less downtime. A target of 99.9 percent allows 8.76 hours a year. A target of 99.99 percent allows 52.6 minutes.
• Parts in series lower the total availability. Copies in parallel raise it, but only when they fail independently.
• A single point of failure is one part that can stop the whole system. Remove it by adding a copy.
• Redundant copies run as active-active or active-passive.
• Copies force a consistency choice. Strong consistency keeps copies identical but lowers availability. Eventual consistency lets copies reach the same data over time.
• The downtime of one failure is the time to detect, decide, and switch. Automatic failover keeps it short.
• Availability = MTBF / (MTBF + MTTR). Faster recovery is often the easiest way to add a nine.
• An SLI is what you measure, an SLO is your target, and an SLA is your promise to customers.

High availability does not come from one tool. It comes from many small decisions: copies in separate places, health checks, automatic failover, safe releases, and a clear target. Start with the target because it tells you how many of these decisions the product really needs.

Practice Questions

Try each question first, then open the answer.

1. A service is down for 72 minutes in a 30-day month. What is its time-based availability?

<details> <summary>Show answer</summary>

About 99.833 percent. A 30-day month has 30 x 24 x 60 = 43,200 minutes. The uptime is 43,200 - 72 = 43,128 minutes. Divide 43,128 by 43,200 and multiply by 100. The result is 99.8333 percent.

</details>

2. Out of 200,000 requests, 500 fail. What is the request-based availability?

<details> <summary>Show answer</summary>

99.75 percent. The number of successful requests is 200,000 - 500 = 199,500. Divide 199,500 by 200,000 and multiply by 100.

</details>

3. Two app servers share one database. Does adding a third app server remove the database as a single point of failure?

<details> <summary>Show answer</summary>

No. Every app server still needs the same database. If the database fails, all three app servers fail with it. The database needs its own copy, like a replica, and its own failover plan.

</details>

4. Course suggestions fail, but lessons remain usable. Which availability technique does this show?

<details> <summary>Show answer</summary>

Graceful degradation. The main action, reading lessons, keeps working while an optional feature, course suggestions, is unavailable.

</details>

5. A replica is ten seconds behind the primary when the primary fails. What could happen if that replica becomes the new primary?

<details> <summary>Show answer</summary>

Some writes may be lost. Updates from the last ten seconds had not reached the replica yet. These can include writes that the old primary already reported to users as successful. This is why some systems wait for a replica to confirm a write before they report success.

</details>