0% completed
Fork, Exec and How Servers Start Workers
On This Page
Fork: One Process Becomes Two
Exec: Running a Different Program
Copy on Write
Prefork Servers
In Production
Using This in an Interview
Key Takeaways
Practice Questions
pid = os.fork()
That single line of Python is called once and returns twice. When it returns, there are two processes running the same program from the same point, with the same variables and the same open files. Gunicorn, Nginx, Postgres, and every Linux shell are built on this one call.
This page explains how a process creates another process and how the new process starts a different program. It also shows why copying a process with hundreds of megabytes of memory takes only a few milliseconds, and how servers use all of this to start their workers.
Fork: One Process Becomes Two
fork is the system call that creates a new process as a copy of the calling process. The new process is the child, and the caller is its parent. The child receives a copy of the parent's memory, its file descriptors, its environment, and its credentials, which are the things listed in What a Process Owns.
The two processes are told apart by the return value. In the parent, fork returns the child's PID. In the child, it returns 0. The code after the call checks that value and takes a different branch in each process.
Exec: Running a Different Program
execve is the system call that replaces the program running in a process with a new program, while the process keeps its PID and its open descriptors. Descriptors marked close-on-exec are the exception, because the kernel closes them during the call. Together, fork and execve are how one program starts another. A shell that runs ls forks itself, and the child immediately calls execve for /bin/ls.
The program forkexec.py does exactly that. The parent forks, the child runs /bin/echo, and the parent waits for it with wait4:
pid = os.fork() if pid == 0: os.execv("/bin/echo", ["echo", "child: now running /bin/echo"]) _, status = os.waitpid(pid, 0)
It ran under strace -f, which follows child processes too, on a cloud virtual machine with 2 CPUs and Linux kernel 7.0. The first column of the trace is the PID that made each call:
Output:
parent 2664: starting a child child: now running /bin/echo parent: child 2665 exited with code 0 2664 execve("/usr/bin/python3", ["python3", "forkexec.py"], <environment>) = 0 2664 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0ca6a78350) = 2665 2664 wait4(2665, <unfinished ...> 2665 execve("/bin/echo", ["echo", "child: now running /bin/echo"], <environment>) = 0 2664 <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2665
The program's own three lines come first, and the trace follows them. Process 2664 is the parent, and the trace starts with the execve that made it a Python process. Its clone call is the fork: Python's os.fork uses the clone system call, and the SIGCHLD flag asks the kernel to notify the parent when the child ends. In the parent, the call returned 2665, the child's PID. Process 2665 then called execve for /bin/echo and became that program, and the parent's wait4 returned when the child exited with code 0.
Copy on Write
Copying hundreds of megabytes on every fork would make it slow. Linux avoids the copy. The kernel manages memory in pages, which are blocks of 4 KB, and it keeps a page table for each process, which records the physical page behind each page of the address space. Copy on write is the technique in which parent and child share the same physical memory pages after fork, and the kernel copies a page only when one of them writes to it. Until then, both page tables point at the same pages, and the kernel marks those pages read-only so that the first write traps into the kernel.
The program cow.py fills 200 MB, forks, and then has the child write one byte into each 4 KB page of the first 50 MB. It reports three numbers from /proc/<pid>/smaps_rollup. Rss is the memory the process uses right now, Shared_Dirty is written memory that it shares with another process, and Private_Dirty is written memory that only it has:
for i in range(0, 50 * MB, 4096): data[i] = 2 # one byte per 4 KB page
Output:
parent before fork: Rss 210 MB Shared_Dirty 0 MB Private_Dirty 203 MB fork took 5.0 ms child after fork: Rss 207 MB Shared_Dirty 203 MB Private_Dirty 0 MB child after writing 50 MB: Rss 207 MB Shared_Dirty 153 MB Private_Dirty 50 MB parent at the end: Rss 210 MB Shared_Dirty 153 MB Private_Dirty 50 MB
Right after fork, the child's Private_Dirty was 0 MB, and 203 MB was shared with the parent. The fork itself took 5.0 ms, which is the time to copy the page tables, not 200 MB of data. After the child wrote to 50 MB, exactly 50 MB became private to the child and 153 MB stayed shared. The parent's numbers changed in the same way, because the original copies of those 50 MB now belonged to the parent alone.
Python reduces this sharing over time, because reading an object also changes its reference count, which is stored in the object itself. Each such change makes the kernel copy a page. Python's garbage collector, which finds and frees unused objects, also writes to the objects it examines. Calling gc.freeze() in the master before forking tells the collector to skip the objects that exist at that moment, which removes that second source of copies but not the first.
Prefork Servers
A prefork server is a server whose master process opens the listening socket and then forks a fixed number of worker processes, which all accept connections from that one socket. Each worker inherits the socket's descriptor through fork, so no worker has to open a socket of its own. Nginx and Gunicorn both work this way.
The demo ran Checkout as a WSGI application under Gunicorn with three workers. WSGI is the standard interface between Python web servers and Python web applications:
Output:
PID PPID COMMAND 2721 2658 gunicorn 2723 2721 gunicorn 2724 2721 gunicorn 2725 2721 gunicorn
The master, process 2721, forked three workers. sudo ss -tlnp then listed a single listening socket on 127.0.0.1:8080, held by all four processes: the master and workers 2723, 2724, and 2725, each through descriptor 3.
Six orders sent at the same time were answered by the workers, and each reply reported the PID of the worker that handled it:
Output:
{"order_id": 1042, "status": "placed", "worker_pid": 2725} {"order_id": 1042, "status": "placed", "worker_pid": 2725} {"order_id": 1042, "status": "placed", "worker_pid": 2724} {"order_id": 1042, "status": "placed", "worker_pid": 2724} {"order_id": 1042, "status": "placed", "worker_pid": 2723} {"order_id": 1042, "status": "placed", "worker_pid": 2724}
Worker 2724 answered three orders, worker 2725 answered two, and worker 2723 answered one. Each worker was waiting on the same socket, and the kernel handed each new connection to one of them.
The master does no request work. Its job is to keep the right number of workers alive. One worker was then killed with SIGKILL:
Output:
PID PPID COMMAND 2721 2658 gunicorn 2724 2721 gunicorn 2725 2721 gunicorn 2751 2721 gunicorn
Worker 2723 disappeared, and a new worker, 2751, took its place within two seconds. Gunicorn's log, with its timestamps removed, records both events:
Output:
[2721] [ERROR] Worker (pid:2723) was sent SIGKILL! Perhaps out of memory? [2751] [INFO] Booting worker with pid: 2751
In Production
| Symptom | What is happening | Command |
|---|---|---|
| Each worker's private memory slowly grows toward the master's size | Shared pages are copied as workers write to them, including reference count changes in Python | grep -E 'Pss|Shared|Private' /proc/<pid>/smaps_rollup for each worker |
| Total memory looks far larger than the machine has | Adding up each worker's RSS from ps counts shared pages several times | Add up Pss instead, which divides each shared page among its users |
| Workers restart, and the log says a worker was killed | A worker exceeded a timeout or a memory limit, and the master replaced it | The server's log, then journalctl -k for memory kills |
| A forked worker hangs forever at startup | The parent had other threads when it forked; a lock held by one of them stays locked in the child | Avoid fork in a process with threads, or fork before starting them |
| Two workers share one database connection | The connection was opened before fork and inherited by every worker | Open connections after forking, for example in a post-fork hook |
The last two rows have the same cause: a forked child gets a copy of everything, including things that were never meant to be shared. Python 3.12 warns when a process with several threads calls fork, for that reason.
Using This in an Interview
A common question is what happens when a process calls fork. A strong answer is: "The kernel creates a child with a copy of the parent's descriptors and a copy of its page tables. The memory itself is shared and marked read-only, and a page is copied only when one side writes to it. That is why forking a process with 200 MB took 5.0 ms in a test."
- "How does a shell run a command?" It forks itself, and the child calls
execvefor the command's program. The parent then waits for the child withwait. - "Why do prefork servers fork before handling requests?" Each worker inherits the listening socket and the loaded code, and the code's pages stay shared, so many workers cost less memory than many separately started programs.
Key Takeaways
forkcreates a child that is a copy of the parent; it returns the child's PID in the parent and 0 in the child.execvereplaces the program in a process and keeps its PID and its descriptors, except those marked close-on-exec.- After
fork, memory is shared with copy on write, so forking 200 MB took 5.0 ms, and the child's private memory grew only by the 50 MB it wrote. - A prefork server opens one listening socket in the master and forks workers that inherit it;
ssshowed four processes holding one socket. - The master's job is to keep workers alive; a killed worker was replaced with a new PID.
- Anything open before
forkis shared by every worker, which is a problem for database connections and for locks held by other threads.
Forked workers are separate processes, each with its own memory. Threads are the other way to run work in parallel, and the kernel treats them in a surprisingly similar way. The next lesson, Processes and Threads From the Kernel's View, shows how.
Practice Questions
Try each question first, then open the answer.
1. A program prints "hello" once and then calls fork twice in a row. How many processes exist at the end, and how many times was "hello" printed?
Four processes, and "hello" was printed once, provided the output was flushed before the first fork. The first fork makes two processes, and each of them calls the second fork, which makes four. The print happened before any fork, so it ran once. If the text was still in an output buffer, though, all four processes hold a copy of that buffer, and "hello" appears four times when they flush it.
2. ps shows a master and eight Gunicorn workers, each with an RSS of 300 MB, on a machine with 2 GB of memory. Is the machine out of memory?
Probably not. RSS counts shared pages in every process that uses them. After fork, the workers share most of their pages with the master, so the real total is far below 9 x 300 MB, which would be 2.7 GB. Adding up Pss from smaps_rollup gives a fair total, because it divides each shared page among the processes that share it.
3. Why did fork of a process with 200 MB of data take only a few milliseconds?
The kernel copied the page tables, not the pages. Both processes pointed at the same physical memory, marked read-only. The child's Private_Dirty was 0 MB right after fork, and it grew only when the child wrote.
4. A Gunicorn worker is killed by the kernel for using too much memory. What happens to the service?
<details> <summary>Show answer</summary>The master notices that the worker ended and forks a new one. Requests that the killed worker was handling fail, but the other workers keep serving, and the listening socket stays open because the master and the other workers still hold it.
</details>5. Checkout's master process opens a database connection at startup, then forks four workers. Orders start receiving each other's query results. What went wrong?
<details> <summary>Show answer</summary>All four workers inherited the same connection. Their descriptors point at one socket, so their queries and replies are mixed on it. The fix is to open the connection in each worker after fork.
Reading Progress
0%
On This Page
Fork: One Process Becomes Two
Exec: Running a Different Program
Copy on Write
Prefork Servers
In Production
Using This in an Interview
Key Takeaways
Practice Questions