Grokking Operating Systems and Networking for Backend Engineers
Vote

0% completed

​

One Request, Step by Step

The Kernel and the Process

Arriving at the Server

Reaching the Code

Leaving the Server

Watching It on a Real Server

In Production

Key Takeaways

Practice Questions

One Request, Step by Step
One Request, Step by Step

A customer presses "Place order" on an online store. The browser sends a small HTTP request, POST /orders with the body {"item": "coffee grinder"}, to a service called Checkout. Checkout reads the order, saves it, and answers with an order number. The code that handles the order is about ten lines long:

def do_POST(self): length = int(self.headers["Content-Length"]) order = json.loads(self.rfile.read(length)) reply = json.dumps({"order_id": 1042, "item": order["item"], "status": "placed"}).encode() self.send_response(201) # ... self.wfile.write(reply)

Those lines run in a fraction of a millisecond. Before the first of them runs, however, the request has already passed through a network card, the kernel's network code, two queues, and the scheduler. After the last line, the reply passes through most of those parts again in the other direction.

This page follows one order through every one of those steps, and then watches the same steps happen on a real Linux server.

The Kernel and the Process

The kernel is the part of the operating system that has full control of the hardware and shares it among the programs running on the machine. Checkout is not the kernel. It is a process, which is a running program together with the memory, open files, and network connections that the kernel has given it.

A process cannot touch the network card or the disk directly. Instead, it asks the kernel to do the work through a system call, which is a request from a process to the kernel such as "read from this connection" or "write these bytes to this file". The kernel does the work and returns a result, and only then does the process continue.

The kernel has four jobs that matter to a backend service:

  • Sharing the CPU. Many processes want to run, and each CPU can run only one at a time.
  • Giving out memory. Each process gets its own memory, and the kernel keeps processes out of each other's memory.
  • Keeping files. The kernel stores file data on disks and keeps recent data in memory.
  • Moving data over the network. The kernel sends and receives packets and turns them into connections.

Every one of these jobs adds a place where a request can wait. The rest of this page finds those places.

Arriving at the Server

The order arrives as packets. A packet is a small unit of data, usually at most about 1,500 bytes, with addresses at the front that say where it came from and where it is going.

A network interface card (NIC) is the hardware that sends and receives packets for a machine. On a cloud server, it is often a virtual card that the cloud provider simulates. When a packet arrives, the NIC copies it into memory that the kernel prepared in advance. It then raises an interrupt, which is a signal from hardware that makes a CPU pause its current work and run a short piece of kernel code.

That kernel code checks the packet's addresses. The destination address belongs to this machine, and the destination port is 8080, so the packet is for whichever process is listening on port 8080. A port is a number from 0 to 65535 that tells the kernel which program a packet is meant for.

A socket is the kernel object that represents one end of a network connection. To the process, a socket is only a small number, such as 3 or 4, that it passes to system calls. Checkout created a listening socket for port 8080 when it started, and that socket accepts new connections.

Before any order data arrives, the browser and the server exchange three packets to open a TCP connection. TCP is the protocol that turns separate packets into a reliable, ordered stream of bytes. The kernel completes this exchange by itself, without running any of Checkout's code. The finished connection is then placed in the accept queue, which is a list, kept for each listening socket, of connections that have finished opening but that the process has not yet picked up.

The order data itself arrives next. The kernel stores it in the connection's receive buffer, which is memory the kernel keeps for each connection where incoming bytes wait until the process reads them.

One request through the machine
One request through the machine

Reaching the Code

While no work was available, Checkout was blocked, which means it was waiting inside a system call and was not using any CPU time. The new connection wakes it up. The kernel marks the process as ready to run, and the scheduler gives it a CPU. The scheduler is the part of the kernel that decides which ready process runs on each CPU next. If every CPU is busy, the woken process waits in line, and no log line records that wait.

Once it is running, Checkout takes three steps in order:

  1. Accept. It takes the connection from the accept queue and receives a new socket number for it.
  2. Read. It asks for the bytes in that connection's receive buffer. The kernel copies them into Checkout's own memory.
  3. Run the handler. The HTTP library parses the bytes, and the ten lines of order code run. This step needs no system call, because the work happens in the process's own memory.

A real handler usually makes more system calls. Saving the order means a request to a database over another connection, and writing a log line means a write to a file. Each of those follows the same path through the kernel, in the outgoing direction first.

Leaving the Server

Checkout writes its reply with a send system call. The kernel copies the reply into the connection's send buffer and returns at once. A send buffer is memory the kernel keeps for each connection, where outgoing bytes wait until the network has carried them and the other side has confirmed that it received them.

That early return is easy to misread. When the send call returns successfully, it means "the kernel has the bytes", not "the customer has the bytes". The kernel's TCP code then splits the reply into packets, hands them to the NIC, and keeps a copy of each one until the browser confirms receipt. If a packet is lost, the kernel sends it again, and Checkout never hears about it.

Finally, Checkout closes the socket, and the kernel finishes the connection with the browser.

Watching It on a Real Server

The same steps can be seen directly. The program checkout.py is a small version of Checkout built with Python's standard HTTP server. It handles one request at a time, so a single process does all the work. It ran on a cloud virtual machine with 2 CPUs and Linux kernel 7.0, and a client on the same machine sent it one order with curl.

Because the client and the server share one machine, the order traveled over the loopback interface. The loopback interface is a network interface inside the kernel that connects a machine to itself, so packets sent to 127.0.0.1 never reach a network card. The NIC and interrupt steps are therefore skipped, but every socket step is the same.

The first command, ss, lists the listening socket. ss is a Linux tool that shows the kernel's sockets and their queues.

Output:

State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess LISTEN 0 5 127.0.0.1:8080 0.0.0.0:* users:(("python3",fd=3))

For a listening socket, Send-Q is the accept queue's limit, called the backlog, which is 5 for Python's simple server. Linux admits one connection beyond the backlog before it treats the queue as full. Recv-Q is the number waiting in the queue right now, which is 0. The listening socket is number 3 inside the process. The order was then sent, and Checkout answered:

Output:

{"order_id": 1042, "item": "coffee grinder", "status": "placed"}

While the order was handled, strace was attached to Checkout. strace is a Linux tool that prints each system call a process makes, with its arguments and its result. The trace was limited to the network calls:

Output:

accept4(3, {sa_family=AF_INET, sin_port=htons(<client port>), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC) = 4 recvfrom(4, "POST /orders HTTP/1.1\r\nHost: 127.0.0.1:8080\r\nUse"..., 8192, 0, NULL, NULL) = 162 sendto(4, "HTTP/1.0 201 Created\r\nServer: BaseHTTP/0.6 Pytho"..., 149, 0, NULL, 0) = 149 sendto(4, "{\"order_id\": 1042, \"item\": \"coffee grinder\", \"st"..., 64, 0, NULL, 0) = 64 shutdown(4, SHUT_WR) = 0 close(4) = 0

Each line matches a step described above:

  • accept4(3, ...) = 4. Checkout took one connection from the accept queue of socket 3. The kernel gave the new connection socket number 4.
  • recvfrom(4, ...) = 162. The kernel copied 162 bytes from the receive buffer: the request line, the headers, and the 26-byte body. Checkout was ready to take up to 8,192 bytes, and the whole request arrived in one read.
  • sendto(4, ...) = 149 and = 64. The server wrote the status line with its headers, and then the body, in two calls. Each call returned the full count at once, because the kernel had placed the bytes in the send buffer.
  • shutdown and close. Checkout told the client that no more data was coming, and then gave socket number 4 back to the kernel.

Between recvfrom and the first sendto, the JSON was parsed and the reply was built. The trace shows nothing there, because that work happened in Checkout's own memory.

In Production

The handler's own timer measures only the middle step. Every other step sits in the kernel, and each one can hold a request without any error appearing in the application's logs.

Where a request can wait
Where a request can wait
Waiting placeIt fills whenWhat you noticeWhere to look
Accept queueThe process picks up connections more slowly than they arriveNew clients wait to connect or time out, while the logs stay quietss -tln: Recv-Q near Send-Q
Waiting for a CPUEvery CPU is busyAll requests slow down a little, and handler times look normalCPU usage and load
Receive bufferThe process reads more slowly than data arrivesUnread bytes pile up for each connectionss -tn: Recv-Q above 0
Send bufferA client or the network is slow to accept dataSend calls stop returning at once, and worker threads stay busyss -tn: Send-Q above 0
Lost packetsThe network drops packetsSome requests take hundreds of milliseconds longerPacket resend counters

A common case shows why this matters. A handler reports 4 ms, while clients report 300 ms at the busiest hour. The gap comes from time spent outside the handler, so the first place to look is the kernel's queues, not the application code.

Key Takeaways

  • The kernel controls the hardware. A process such as Checkout asks it for every network and file operation, and for more memory, through system calls.
  • A request arrives at the NIC, passes through the kernel's network code, and waits in the accept queue and then the receive buffer before the process reads it.
  • A blocked process uses no CPU. When work arrives, the scheduler must still give it a CPU, and that wait appears in no log.
  • A send call returns when the kernel has the bytes, not when the client has them.
  • strace shows each system call a process makes, and ss shows the kernel's sockets and their queues.
  • A handler's timer misses every wait that happens in the kernel, so client time and handler time can differ greatly.

Each step on this path costs some time, and the costs differ enormously from one step to the next. The next lesson, Where the Time Goes in a Request, measures them on the same kind of server and builds a time budget for one order.

Practice Questions

Try each question first, then open the answer.

1. In the trace, sendto returned 64 for the body of the reply. Does that mean the customer's browser has received the order number?

<details> <summary>Show answer</summary>

No. It means the kernel has copied 64 bytes into the connection's send buffer. The kernel still has to split the bytes into packets, send them, and wait for the browser to confirm receipt. If a packet is lost, the kernel sends it again without telling Checkout.

</details>

2. ss -tln shows Recv-Q 6 and Send-Q 5 on Checkout's listening socket. What does this mean, and what do new customers experience?

<details> <summary>Show answer</summary>

The accept queue is full. Six finished connections are waiting, which is as many as Linux allows for a backlog of five, so Checkout is not picking up connections as fast as they arrive. New customers see slow connections or connection timeouts, while Checkout's logs show nothing unusual, because those connections never reach its code.

</details>

3. Checkout's handler timer reports 3 ms per order, but the customers' browsers report 400 ms at the busiest hour. Name two places outside the handler where the extra time can be spent.

<details> <summary>Show answer</summary>

Any two of these: the accept queue, the wait for a CPU, the receive buffer, the send buffer, and resent packets. At the busiest hour, the accept queue and the wait for a CPU are the most likely, because both fill when more work arrives than the process can start.

</details>

4. strace printed nothing between recvfrom and the first sendto, even though the handler parsed JSON and built a reply during that time. Why?

<details> <summary>Show answer</summary>

That work happened in Checkout's own memory, so it needed no system call. strace shows only the requests a process makes to the kernel. Parsing and building the reply are ordinary work on data that the process already holds.

</details>

5. Python's server sent the reply's headers and body with two separate sendto calls. Why might combining them into one call matter for a service that handles thousands of requests per second?

<details> <summary>Show answer</summary>

Each system call has a cost, and that cost grows with the request rate. One call instead of two halves that cost for each reply. Two small writes can also leave as two separate packets instead of one, which adds work for the kernel, the network, and the client.

</details>

Reading Progress

0%


Vote for new content

On This Page

The Kernel and the Process

Arriving at the Server

Reaching the Code

Leaving the Server

Watching It on a Real Server

In Production

Key Takeaways

Practice Questions