0% completed
User Mode, Kernel Mode and System Calls
On This Page
Two Modes of the CPU
What a System Call Is
What a Crossing Costs
Many Small Calls or Few Large Ones
Checkout's System Calls
In Production
Using This in an Interview
Key Takeaways
Practice Questions
A small C program tried to do two things that the kernel does every second: switch off hardware interrupts, and read from kernel memory. Both attempts ended the same way, with the program killed at once. A third attempt, which asked the kernel for a small piece of information through the official route, finished normally:
Output:
disable interrupts (cli): killed by signal 11 (Segmentation fault) read a kernel address: killed by signal 11 (Segmentation fault) ask the kernel (getppid): finished normally
Checkout runs under the same rules. It cannot touch the network card, the disk, or another process's memory, so it asks the kernel for each of those things, and every request costs a little time.
This page explains where that boundary between a process and the kernel comes from, what happens when a process crosses it, and how much each crossing costs on a real server.
Two Modes of the CPU
The boundary is built into the processor itself. User mode is the restricted state in which the CPU runs ordinary programs, and kernel mode is the unrestricted state in which it runs the kernel. On x86 processors, these two states are called ring 3 and ring 0.
In user mode, the CPU refuses a set of privileged instructions. cli, which switches off interrupts, is one of them. Instructions that change the memory map or talk directly to devices are others. The kernel's own memory is also marked as reachable only in kernel mode, so a user-mode read of a kernel address fails.
When a user-mode program breaks one of these rules, the CPU stops it and hands control to the kernel. The kernel then sends the program a signal, which is a notice from the kernel to a process, and signal 11 (SIGSEGV) ends the program by default. That is what happened to the first two attempts above.
The rule exists so that one faulty or hostile program cannot crash the machine or read another program's data. It also means that every useful thing a server does, such as reading a socket or writing a log line, needs the kernel's help.
What a System Call Is
A system call is a controlled entry into the kernel: the process names the service it wants with a number, and the CPU switches to kernel mode at an entry point that only the kernel chooses. The process cannot jump to an arbitrary place in the kernel. It can enter only through this one entry point.
On x86-64 Linux, the steps are always the same:
- The process puts the system call's number in a register, which is a small storage slot inside the CPU, and puts its arguments in other registers. For example,
writeis number 1. Its arguments are a file descriptor, which is the small number a process uses to name an open file or socket, then a memory address and a length. - The process runs the
syscallinstruction. The CPU switches to kernel mode and jumps to the kernel's entry code. - The kernel saves the process's registers and checks every argument. A pointer into kernel memory, for example, is refused with the error
EFAULT. - The kernel does the work, puts the result in a register, and returns to user mode at the next instruction of the process.
Programs rarely run that instruction themselves. Python's os.write, Java's FileOutputStream.write, and Go's os.File.Write all end in the same write system call, through the C library or the language runtime. Linux on x86-64 offers a few hundred system calls in total, and a typical web service uses a few dozen of them.
What a Crossing Costs
The same C program timed a plain function call, two system calls, and one call that looks like a system call but is not. It ran on a cloud virtual machine with 2 CPUs and Linux kernel 7.0. A nanosecond (ns) is one billionth of a second, so 1 µs equals 1,000 ns.
n = 2000000; t0 = now_ns(); for (int i = 0; i < n; i++) syscall(SYS_getppid); printf("system call (getppid): %7.1f ns\n", (now_ns() - t0) / n);
Output:
plain function call: 1.2 ns system call (getppid): 144.7 ns clock_gettime as a system call: 214.8 ns clock_gettime through the vDSO: 20.4 ns
getppid does almost no work; it only returns the process ID of the parent. Its 145 ns is therefore almost entirely the cost of the crossing, which is about 120 times the cost of a function call. The time goes to switching modes, saving and restoring registers, checking arguments, and the extra protective steps that many processors now need on every entry into the kernel. Where the Time Goes in a Request measured 0.22 µs for the same call on a different machine. The cost changes with the processor and its protections, but it stays at a few hundred nanoseconds or less.
The last two lines show a shortcut. The vDSO (virtual dynamic shared object) is a small piece of kernel code that Linux maps into every process, so that a few read-only calls, such as reading the clock, run in user mode without a crossing. Reading the clock through the vDSO took 20 ns, against 215 ns for the same call made as a real system call. Services read the clock constantly for timeouts, logs, and metrics, so this shortcut matters.
Many Small Calls or Few Large Ones
The cost per call matters most when a program makes calls that each move very little data. The program wrote 1 MB to /dev/null twice: first one byte per write, then 64 KB per write. The shell's time command reports user time, which is CPU time spent in the program's own code, and sys time, which is CPU time the kernel spent working for it:
Output:
1048576 write calls of 1 bytes real 0.257 s user 0.057 s sys 0.200 s
Output:
16 write calls of 65536 bytes real 0.001 s user 0.000 s sys 0.001 s
The data was the same, but the first version made 65,536 times as many calls and took more than 100 times as long. (The second time is rounded to 0.001 s, so a more exact ratio cannot be read from it.) Most of the first version's time, 0.200 s of 0.257 s, was sys time, which means crossings rather than useful work. This is why buffered I/O exists: Python's file objects, Java's BufferedOutputStream, and Go's bufio collect small writes in the process's memory and pass them to the kernel in large pieces.
Checkout's System Calls
The small Checkout service from One Request, Step by Step handled 100 orders while strace -c counted its system calls. The -c option prints a summary table instead of each call:
Output:
% time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 34.98 0.002055 10 200 sendto 14.11 0.000829 8 102 poll 8.04 0.000472 4 100 close 5.92 0.000348 3 100 recvfrom 22.23 0.001306 13 100 shutdown 5.12 0.000301 3 100 getsockname 9.58 0.000563 5 100 accept4 0.00 0.000000 0 1 restart_syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.005874 7 803 total
That is eight system calls per order: poll to wait for a connection, accept4, getsockname, recvfrom, two sendto calls, shutdown, and close. The times in the table are much larger than 145 ns for two reasons. Calls such as sendto and shutdown do real work, because the kernel builds and delivers packets during the call. And strace stops the process twice for every call it records, which adds its own cost. The counts are reliable, but the timings mix real work with strace's overhead, so they are a poor measure of any one call.
In Production
Crossings rarely slow down one request, but they decide how much CPU a busy service needs. The symptom is CPU time that goes to the kernel instead of to the application.
| Symptom | How it looks | Command that shows it |
|---|---|---|
| Too many small writes | High system CPU while traffic is normal; thousands of write or sendto calls with tiny sizes | strace -c -p <pid> for a few seconds, then strace -e trace=write -p <pid> |
| Unbuffered logging | One write per log line, often several per request | strace -e trace=write -p <pid> |
| Clock reads that cross into the kernel | Many clock_gettime calls in strace -c, which normally never appear there | cat /sys/devices/system/clocksource/clocksource0/current_clocksource |
| High system time overall | sy in top well above us | top, then strace -c on the busiest process |
The clock row needs a word of explanation. The vDSO shortcut works only with clock sources that user code may read directly. On some virtual machines, the kernel uses a clock source that does not allow this, and every clock read becomes a real system call. Because strace cannot see calls that stay in the vDSO, a sudden column of clock_gettime calls in its summary is the sign.
Use strace for seconds, not minutes, on a production process. Every traced call stops the process twice, so a busy service can become several times slower while it is attached.
Using This in an Interview
A common question asks what a system call is and why it costs more than a function call. A short answer is: "A system call switches the CPU from user mode to kernel mode at an entry point the kernel controls, and the kernel checks every argument before it does the work. On a cloud server that crossing costs about 150 ns, against about 1 ns for a function call."
- "How do programs reduce the number of system calls?" They buffer small reads and writes in user memory and pass them to the kernel in large pieces. A few calls, such as reading the clock, avoid the crossing completely through the vDSO.
- "Why can a program not simply run kernel code itself?" The CPU refuses privileged instructions and kernel memory in user mode. A program that tries is stopped, and the kernel sends it
SIGSEGV.
Key Takeaways
- The CPU runs programs in user mode and the kernel in kernel mode. Privileged instructions and kernel memory are refused in user mode, and a program that tries them receives
SIGSEGV. - A system call is the one controlled way into the kernel: a call number and arguments in registers, the
syscallinstruction, argument checks, the work, and a return. - On this server, a system call cost about 145 ns, while a function call cost about 1 ns.
- The vDSO lets a few read-only calls, such as reading the clock, run without a crossing, in about 20 ns.
- One megabyte written one byte per call took 0.257 s, mostly in the kernel, while the same megabyte in 64 KB calls took 0.001 s.
strace -ccounts a process's system calls reliably, but its timings includestrace's own overhead.
Every system call names things that the process owns, such as file descriptor 4 or a memory address. The next lesson, What a Process Owns, opens a running Checkout process in /proc and lists those things one by one.
Practice Questions
Try each question first, then open the answer.
1. A Python service writes every log line with os.write directly to a file descriptor, and it logs five lines per request at 4,000 requests per second. How many system calls per second do the logs cost, and what is the usual fix?
About 20,000 write calls per second, one per log line. At roughly 150 ns or more each, that is several milliseconds of kernel time every second, plus the work of each write. The usual fix is a buffered writer, which collects lines in memory and writes them in large pieces, or a logging library that does the same.
2. In the test above, getppid cost about 145 ns, although it only returns one number. Where does that time go?
Almost all of it is the crossing itself. The CPU switches to kernel mode, the kernel saves the process's registers, runs its entry code and protective steps, and then switches back. The work of finding the parent's process ID is a tiny part of the total.
</details>3. strace -c on a busy service shows 200,000 clock_gettime calls in ten seconds. Why is that surprising, and what would you check?
Clock reads normally go through the vDSO and never appear in strace. Seeing them means each read is a real system call, which usually happens when the machine's clock source does not allow user-mode reads. Check /sys/devices/system/clocksource/clocksource0/current_clocksource and compare it with the clock sources the cloud provider recommends.
4. A colleague runs strace -c on Checkout and reports that shutdown takes 13 µs per call, so it must be a bottleneck. What is wrong with that conclusion?
The timings in strace -c are inflated by strace itself. It stops the process twice for every call it records, so every call looks slower than it really is. shutdown does some real work too, because it sends a packet that ends the connection, but the table cannot separate that work from the overhead. The call counts are reliable, but the timings should not be used to rank calls without a lower-overhead tool.
5. Why does a program that executes cli get SIGSEGV and not a different error such as "permission denied"?
Because the CPU, not the kernel's permission checks, refused the instruction. In user mode, the processor treats cli as a protection fault and passes control to the kernel. The kernel's response to that fault is to send the program signal 11, which ends it. "Permission denied" is an error that a system call returns, and no system call was made.
Reading Progress
0%
On This Page
Two Modes of the CPU
What a System Call Is
What a Crossing Costs
Many Small Calls or Few Large Ones
Checkout's System Calls
In Production
Using This in an Interview
Key Takeaways
Practice Questions