Grokking Operating Systems and Networking for Backend Engineers
Vote

0% completed

​

Where the Time Goes in a Request

The Units

How the Costs Were Measured

Reading the Numbers

Counting Round Trips in One HTTPS Request

A Time Budget for One Order

Where Small Costs Still Matter

In Production

Using This in an Interview

Key Takeaways

Practice Questions

Where the Time Goes in a Request
Where the Time Goes in a Request

On a typical cloud server, a system call takes about 0.2 microseconds. A network round trip from Virginia to Singapore takes about 222 milliseconds, which is roughly a million times longer. Both kinds of cost occur inside an ordinary request, together with context switches, page faults, disk writes, and TLS handshakes that fall somewhere between them.

When a request is slow, these differences decide where to look first. A slow request is almost never slow because of one extra system call, and it is often slow because of one extra round trip.

This page measures each of these costs on a real Linux server and then uses the measurements to build a time budget for one Checkout order.

The Units

A millisecond (ms) is one thousandth of a second, and a microsecond (µs) is one millionth of a second. One millisecond is therefore 1,000 microseconds. Kernel operations are usually measured in microseconds, while disks and networks are usually measured in milliseconds.

How the Costs Were Measured

Two small programs ran on a cloud virtual machine with 2 CPUs and Linux kernel 7.0, in Microsoft Azure's East US region in Virginia. The first program, written in C, measured the kernel costs. C was used because Python adds its own work to every pass through a loop, and that work would be a large share of a cost this small. Each operation ran tens of thousands of times or more, and the program printed the average:

int n = 1000000; double t0 = now_ns(); for (int i = 0; i < n; i++) syscall(SYS_getppid); printf("system call (getppid): %6.2f us\n", (now_ns() - t0) / n / 1000);

The second program, written in Python, measured disk, loopback, TLS, and network costs, which are large enough that Python's own overhead matters little. The loopback figures still include some Python work, so they are upper limits. Each figure is the median of between 15 and 5,000 tries. For example, the disk measurement writes 4 KB and then times fsync 200 times:

for _ in range(200): os.write(fd, block) start = time.perf_counter() os.fsync(fd) times.append(time.perf_counter() - start)

Output:

system call (getppid): 0.22 us context switch (pipe ping-pong): 3.62 us page fault (first touch of a page): 1.73 us read 4 KB from the page cache: 0.94 us fsync after a 4 KB write: 0.30 ms loopback round trip, open connection: 0.040 ms loopback round trip, new connection: 0.145 ms TLSv1.3 handshake on loopback: 0.96 ms TCP connect to us-east-1 (Virginia) 3.5 ms TCP connect to us-west-2 (Oregon) 78.2 ms TCP connect to ap-southeast-1 (Singapore) 221.9 ms

Reading the Numbers

The results fall into three groups.

Eleven costs on one scale
Eleven costs on one scale

The first group costs about a microsecond each. A system call took 0.22 µs. A context switch, which is the kernel stopping one process on a CPU and starting another, took 3.6 µs. That figure also includes two small pipe calls per switch, so it is an upper limit. A page fault took 1.7 µs; it happens when a process first touches memory that the kernel has promised but not yet prepared, so the kernel pauses the process and sets up the page. Reading 4 KB of a file took 0.94 µs, because the data was already in the page cache, which is memory where the kernel keeps recently used file data.

The second group costs tens of microseconds to about a millisecond. A request and reply over an open loopback connection took 40 µs. Opening a new connection for each request raised that to 145 µs, because the connection setup adds packets and system calls. A TLS 1.3 handshake took 0.96 ms even though no distance was involved; that time is CPU work for the key exchange and the certificate signature. fsync is the system call that writes a file's changed data to the disk and waits until the disk confirms it. On this server's disk, it took 0.30 ms, and a database usually waits for one fsync each time it commits.

The third group costs many milliseconds, and the cause is distance. Opening a TCP connection takes one round trip, so the connect times measure the round trip to each region. The AWS region in Virginia is near this server, and the round trip took 3.5 ms. Oregon took 78 ms, and Singapore took 222 ms.

Light in an optical fiber travels about 200 km per millisecond. Virginia and Singapore are about 15,500 km apart in a straight line, so a round trip cannot take less than about 155 ms, and real cable routes are longer than a straight line. No change to software can shorten that time. The only remedies are to move the two ends closer together or to make fewer round trips.

These numbers vary from machine to machine. Across six runs on different cloud machines, a system call took between 0.13 and 0.34 µs, a context switch between 1.6 and 3.6 µs, and fsync between 0.25 and 0.41 ms. None of them changed by a factor of ten, and the order of the groups never changed.

A round trip between two servers inside one data center was not measured here. It is usually well under a millisecond, and this page assumes 0.5 ms. Running ping from one of your servers to another gives the real figure for your network.

Counting Round Trips in One HTTPS Request

The tool curl can split a request into stages. The same server fetched a page from the AWS region in Singapore, and curl printed the time at which each stage finished, counted from the start:

Output:

dns 0.005758s connected 0.226867s tls done 0.671319s first byte 0.892073s

The DNS lookup finished after 6 ms. The TCP connection then took one round trip, about 221 ms. The TLS handshake then took 444 ms, which is two round trips, and the first byte of the reply arrived one round trip after that. In total, the request needed four round trips to Singapore and took about 0.9 seconds.

TLS 1.3 normally needs only one round trip, so the second one needed an explanation. The handshake messages, printed with curl -v, showed two hello messages from each side. The client had offered a key for one kind of key exchange (X25519), while this server prefers another kind (P-256). The server therefore asked the client to start again with the other kind, and that request cost a full round trip. When curl was told to offer P-256 first, the handshake took one round trip:

Output:

dns 0.068522s connected 0.289721s tls done 0.513873s first byte 0.736255s

TLS now took 224 ms instead of 444 ms. A single configuration detail changed the cost of every new connection to this server by about 220 ms.

Four round trips to Singapore
Four round trips to Singapore

A Time Budget for One Order

Suppose Checkout runs in Virginia, and a customer in Singapore places an order over a new HTTPS connection. Checkout makes three database queries in its own data center, commits the order with one fsync, and calls a payment provider in Oregon over a connection that is already open. Assume the TLS handshake needs one round trip, and that handling the order takes about 40 system calls and 4 context switches.

StepCountCost eachTotal
Customer: TCP connection1 round trip222 ms222 ms
Customer: TLS 1.3 handshake1 round trip222 ms222 ms
Customer: request and reply1 round trip222 ms222 ms
Payment provider call1 round trip78 ms78 ms
Database queries3 round trips0.5 ms1.5 ms
TLS work on the CPU10.96 msabout 1 ms
Commit fsync10.30 ms0.3 ms
Context switches43.6 µs0.014 ms
System calls400.22 µs0.009 ms

The total is about 747 ms, and four long round trips account for 744 ms of it, which is more than 99 percent. The kernel work in the last two rows adds up to less than a tenth of a millisecond.

The budget also shows which changes are worth making. When the customer's browser reuses its connection for the next order, the first two rows disappear and the order takes about 300 ms. If Checkout opened a new connection to the payment provider for every order, it would add two more round trips to Oregon, or about 156 ms.

Where Small Costs Still Matter

Microsecond costs do not change the time of one request, but they decide how many requests a CPU can handle. Suppose Checkout serves 20,000 requests per second. Its system calls then cost 20,000 x 40 x 0.22 µs, which is about 0.18 seconds of CPU time every second, or 18 percent of one CPU. Its context switches cost another 29 percent. Halving the number of system calls per request does not make any one order faster, but it can save a whole CPU across a busy fleet.

In Production

Count round trips first, then disk flushes, and only then everything else. Most surprising latency comes from a round trip that nobody expected: a new TLS connection for each call to another service, a retry, or a query that runs once per item in a loop.

These costs also work as a quick check on a trace. If a call between two services in one data center takes 40 ms, distance cannot explain it, because the round trip is well under a millisecond. The cause is more likely a queue, a new connection, or slow work on the other side.

ToolWhat it measures
ping <host>The round trip time to another machine
curl -w with the timing fieldsDNS, connect, TLS, and first byte for one HTTPS request
strace -c -p <pid>How many system calls a process makes, and of which kinds

Using This in an Interview

Estimates of latency are a common part of design discussions, and the useful skill is to count round trips: "The client is in Singapore and the service is in Virginia, so each round trip is about 220 ms. A new HTTPS connection needs three round trips before the first byte, so the first request takes at least two-thirds of a second, and later requests on the same connection take about 220 ms."

Two follow-up questions are common:

  • "How would you make the first request faster?" End the TCP and TLS connection at a server near the customer. Those two round trips then cross a short distance, and only the request itself travels the full distance.
  • "Would a faster CPU help?" Very little. The CPU work for one request is about a millisecond, while the round trips are hundreds of milliseconds.

Key Takeaways

  • The costs inside one request span about six orders of magnitude, from 0.22 µs for a system call to 222 ms for a round trip to Singapore.
  • System calls, context switches, page faults, and page cache reads each cost about a microsecond.
  • Loopback round trips, TLS handshakes, and fsync cost from tens of microseconds to about a millisecond.
  • A round trip across a continent or an ocean costs tens to hundreds of milliseconds, and only distance or fewer round trips can reduce it.
  • A new HTTPS connection costs at least three round trips before the first byte, and a key type the server does not prefer added a fourth.
  • Microsecond costs rarely change one request's latency, but at high request rates they decide how much CPU a service needs.

The cheapest cost on this page, the system call, is also the most frequent one, because every network and file operation goes through it. That boundary between a process and the kernel is the subject of the next lesson, User Mode, Kernel Mode and System Calls.

Practice Questions

Try each question first, then open the answer.

1. A service in Virginia calls a service in Oregon five times, one call after another, for each user request, over connections that are already open. Using this page's measurements, what is the least time those calls can add to each request, and how can it be reduced?

<details> <summary>Show answer</summary>

At least about 390 ms, because each call needs one round trip of about 78 ms. Making the five calls at the same time reduces the wait to about one round trip, if the calls do not depend on each other. Combining them into one call has the same effect. Moving the two services into the same region removes most of the cost.

</details>

2. A team proposes rewriting a service in Virginia in a faster language because its requests take about 1.4 seconds. A trace shows two calls to a server in Singapore, each over a new HTTPS connection. What does this page suggest instead?

<details> <summary>Show answer</summary>

Reuse the connection to Singapore. Two calls of three round trips each, at about 222 ms per round trip, explain about 1.3 seconds. Each new HTTPS connection needs a TCP round trip and a TLS round trip before the request's own round trip, so a pooled connection removes two round trips from each call. The CPU work for a request is about a millisecond, so a faster language would change almost nothing.

</details>

3. Why was the system call cost measured with a C program instead of with Python?

<details> <summary>Show answer</summary>

Because Python adds its own work to every pass through a loop, and that work is a large share of a 0.22 µs cost. C adds almost nothing. Disk, TLS, and long-distance network costs are hundreds to a million times larger, so Python's overhead is a small part of them.

</details>

4. A database commits each transaction with one fsync, which takes 0.30 ms on its disk. If it commits one transaction at a time, about how many commits per second can it make? How can a database do better without a faster disk?

<details> <summary>Show answer</summary>

About 3,300 commits per second, because 1 second divided by 0.30 ms is about 3,333. A database can do better by writing several transactions' changes and then calling fsync once for all of them. One wait then confirms many commits.

</details>

5. In the curl output, the TLS handshake to Singapore took 444 ms, although TLS 1.3 normally needs one round trip. What happened, and how was it shown?

<details> <summary>Show answer</summary>

The server asked the client to restart the handshake with a different kind of key exchange. The client offered X25519 first, and the server prefers P-256, so the handshake took a second round trip. When curl offered P-256 first, the handshake took 224 ms, which is one round trip.

</details>

Reading Progress

0%


Vote for new content

On This Page

The Units

How the Costs Were Measured

Reading the Numbers

Counting Round Trips in One HTTPS Request

A Time Budget for One Order

Where Small Costs Still Matter

In Production

Using This in an Interview

Key Takeaways

Practice Questions