0% completed
How Linux Runs a Process
On This Page
Fork and Exec
Process States
Signals
Exit Codes
Zombies and Orphans
PID 1 and systemd
PID 1 Inside a Container
Using This in an Interview
Key Takeaways
Practice Questions
ps -eo pid,ppid,stat,args PID PPID STAT COMMAND 1 0 Ss /usr/lib/systemd/systemd 812 1 Ss nginx: master process /usr/sbin/nginx 814 812 S nginx: worker process 3307 1 Ssl /usr/bin/python3 /opt/worker/run.py 3412 3307 Z [gzip] <defunct> 3420 3307 D /usr/bin/python3 /opt/worker/run.py
Each line is one process on a server. Process 3412 has already finished but still appears in the list. Process 3420 is waiting for the disk, and it will not respond even to kill -9 until that wait ends.
Both facts follow from rules that Linux applies to every process. The sections below explain how a process starts, waits, receives signals, and ends, and why those rules matter inside a container.
Fork and Exec
A process is a running instance of a program, with its own memory, open files, and process ID (PID). Every process except the first has a parent, shown in the PPID column, so nginx master 812 started worker 814.
Linux starts a new program in two steps: fork copies the current process, and exec replaces the program inside the copy. After fork, the parent and the child run the same code, but the child has a new PID. The fork call returns the child's PID to the parent and 0 to the child, so each copy knows its role.
When you type ls in a shell, the shell forks, and the child calls exec to load the ls program. Exec keeps the PID and the open files, so the child still writes to the same terminal. The shell then waits for the child to finish and reads its exit code.
A program may fork without calling exec. Process 3420 is a copy of worker 3307 that never ran exec, so it shows the same command line.
Process States
The first letter of the STAT column shows the state of the process. Later letters add detail, such as l for a process with several threads.
| Letter | State | Meaning | What many of them suggest |
|---|---|---|---|
| R | Running | On a CPU, or ready and waiting for one | High CPU demand |
| S | Interruptible sleep | Waiting for an event, such as network data | Normal for most processes |
| D | Uninterruptible sleep | Waiting inside the kernel, usually for disk or network storage | Slow or unreachable storage |
| Z | Zombie | Exited, but the parent has not collected its exit status | A parent with a bug |
| T | Stopped | Paused by a signal or a debugger | Someone paused it on purpose |
A process in D state acts on no signal until the kernel operation returns. A long D state therefore points to storage that has stopped responding, such as an unreachable network file system.
Signals
A signal is a small numbered message that the kernel delivers to a process to report an event or to request that it stop. A process can install a handler, a function that runs when a particular signal arrives. Without a handler, the signal's default action applies, and for the four signals below that action ends the process.
| Signal | Number | Usual source | Can it be caught? |
|---|---|---|---|
| SIGHUP | 1 | A closed terminal; many services treat it as "reload configuration" | Yes |
| SIGINT | 2 | Ctrl+C in a terminal | Yes |
| SIGKILL | 9 | kill -9, or the kernel when memory runs out | No |
| SIGTERM | 15 | kill with no option, systemd, or a container stop | Yes |
SIGTERM is a request: a well-written service catches it, stops accepting new work, finishes its current requests, and exits. SIGKILL ends the process at once, so its open requests fail.
Exit Codes
An exit code is a number from 0 to 255 that a process returns when it ends, and 0 means success.
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | A general failure chosen by the program |
| 126 | The command was found but could not run, often because it is not executable |
| 127 | The command was not found |
| 128 plus n | The process was ended by signal n |
Exit code 137 is 128 plus 9, so SIGKILL ended the process, for example because memory ran out. Exit code 143 is 128 plus 15, so SIGTERM ended a process that had no handler for it.
Zombies and Orphans
A zombie is a process that has exited but whose parent has not yet collected its exit status. The kernel frees the zombie's memory and files but keeps its PID and its entry in the process table until the parent calls wait. Collecting the status in this way is called reaping, and in the opening output, worker 3307 has not reaped process 3412.
One zombie costs almost nothing, but a parent that never reaps can leave thousands, each holding a PID. A zombie cannot be killed because it is no longer running, so you fix the parent or end it.
An orphan is a running process whose parent exited first. The kernel makes PID 1 its new parent, or a nearer ancestor that has asked to adopt orphans (a subreaper), and the new parent reaps it later. For this reason, ending a zombie's parent also clears the zombie, because its new parent reaps it.
PID 1 and systemd
PID 1 is the first process outside the kernel (a user-space process) that the kernel starts, and on most current distributions it is systemd. systemd starts and restarts services and reaps orphans. To stop a service, it sends SIGTERM and, after a timeout, SIGKILL.
The kernel treats PID 1 in one unusual way. It does not apply a default action to PID 1, so any signal for which PID 1 has no handler is ignored. A SIGTERM sent to such a PID 1 does nothing at all.
PID 1 Inside a Container
A container's main process is PID 1 in the container's own PID namespace, so the same rule applies to it. A PID namespace is a separate set of process numbers that the kernel gives the container. To stop the container, the container runtime, the program that starts and stops containers, sends SIGTERM, waits for a grace period, and then sends SIGKILL. Docker waits 10 seconds by default, and Kubernetes waits 30 seconds unless the pod sets another value.
If the application has no SIGTERM handler, the kernel ignores the signal, so nothing happens until the grace period ends. SIGKILL from outside the container is the one exception that the kernel still delivers, so it ends the container with exit code 137, and its open requests fail.
The delay also slows deployments. Suppose that a deployment tool stops 10 containers one at a time and waits for each to exit. The stops take 10 x 30, or 300 seconds, instead of 10 x 2, or 20 seconds, so every deployment is 280 seconds slower.
A shell can cause the same problem. If the container starts the application through sh -c, the shell becomes PID 1 and may not pass SIGTERM to its child. Three fixes are common: replace the shell with the application by using exec, add a SIGTERM handler, or run tini, a small program that runs as PID 1, passes signals on, and reaps zombies. The lesson How Containers Work applies these fixes to container images.
Using This in an Interview
"What happens when you run kill on a process?" tests the mechanism behind a familiar command. Answer with the signals, then the cases where they fail.
"By default, kill sends SIGTERM, signal 15, which the process can catch to finish its requests and exit cleanly. If it does not exit, we send SIGKILL, signal 9, which cannot be caught, and the shell then reports exit code 137. A process in D state will not act even on SIGKILL until its I/O returns. A zombie has already exited, so its parent must reap it."
Expect two follow-up questions:
- "How do you remove a zombie?" Make the parent call wait, or end the parent so that PID 1 becomes the new parent and reaps the zombie.
- "Why does my container take 30 seconds to stop?" Its PID 1 probably has no SIGTERM handler, so the runtime waits for the grace period and then sends SIGKILL.
Key Takeaways
- Fork copies a process, and exec replaces its program while keeping the PID.
- A process in D state cannot act on signals until its I/O returns.
- SIGTERM, SIGINT, and SIGHUP can be caught, but SIGKILL cannot.
- An exit code above 128 usually means that a signal ended the process, so 137 means SIGKILL.
- PID 1 ignores signals that it does not handle, so a container's main process must handle SIGTERM.
The next lesson, Memory, CPU, and Load, explains what these processes consume and why a server can show a high load while its CPUs are idle.
Practice Questions
1. A container exits at once with exit code 127. What is the most likely cause?
<details><summary>Show answer</summary>The command that the container tried to run was not found. Code 127 is below 128, so no signal was involved. Check the entry command for a spelling mistake, and check that the program exists in the image and is on the search path.
</details>2. A process has been in D state for 10 minutes, and kill -9 has no effect. What should you investigate?
Investigate the storage that the process is waiting for. A process in D state acts on no signal until its kernel operation returns, so check the kernel log for disk errors and any network file system for reachability.
</details>3. A server shows 4,000 zombie processes, all with the same parent. What should you change?
<details><summary>Show answer</summary>Change the parent so that it reaps its children. The parent must call wait for each child, often from a handler for the signal that reports a finished child. Restarting the parent clears the existing zombies, because PID 1 becomes their parent and reaps them.
</details>Reading Progress
0%
On This Page
Fork and Exec
Process States
Signals
Exit Codes
Zombies and Orphans
PID 1 and systemd
PID 1 Inside a Container
Using This in an Interview
Key Takeaways
Practice Questions