Grokking Operating Systems and Networking for Backend Engineers
Vote

0% completed

​

Signals, Graceful Shutdown and Exit Codes

What a Signal Is

Stopping Without a Handler

Graceful Shutdown

The Grace Period and SIGKILL

SIGHUP and Reloading

Exit Codes

Zombies and Orphans

In Production

Using This in an Interview

Key Takeaways

Practice Questions

Signals, Graceful Shutdown and Exit Codes
Signals, Graceful Shutdown and Exit Codes

A deploy restarts Checkout while one customer's order is in the middle of its payment call. The customer's browser receives an empty reply. Checkout's log ends with the line "order received, calling the payment provider", and nothing follows it, not even an error.

The order was not lost because of a bug in the order code. It was lost because of what Checkout did, or failed to do, when the deploy tool asked it to stop. That request arrives as a signal, and the same mechanism also tells a server to reload its settings and tells a parent that its child has ended.

This page explains signals, shows a shutdown that loses an order and one that does not, and then explains the exit codes, zombies, and orphans that every stopped process leaves behind.

What a Signal Is

A signal is a short notice that the kernel delivers to a process, identified only by a number, which interrupts whatever the process is doing. Another process sends one with the kill system call, and the kernel sends some itself, such as SIGSEGV after a memory fault.

Each signal has a default action, which is usually to end the process. A process can replace that action with a signal handler, which is a function the process registers to run when the signal arrives, or it can ignore the signal. Two signals allow neither: SIGKILL and SIGSTOP always take effect.

SignalNumberUsual meaningDefault actionCan be handled
SIGHUP1Reload settings (servers)End the processYes
SIGINT2Ctrl+C in a terminalEnd the processYes
SIGKILL9Stop immediatelyEnd the processNo
SIGPIPE13Write to a closed connectionEnd the processYes
SIGTERM15Please stopEnd the processYes
SIGCHLD17A child process endedIgnoreYes

Stopping Without a Handler

The demo ran a version of Checkout whose payment call takes 2 seconds, on a cloud virtual machine with 2 CPUs and Linux kernel 7.0. A client sent an order, and half a second later, while the payment call was still running, Checkout received SIGTERM. In this test, Checkout had no handler for it. (It ran after the graceful test below, so its settings file already held a payment timeout of 5,000 ms.)

Output:

0.00s listening on port 8080 0.94s order received, calling the payment provider (timeout 5000 ms) $ echo $? (exit code of Checkout) 143 $ curl output curl: (52) Empty reply from server (HTTP 000)

The default action ended the process at once, in the middle of the order. The shell reported exit status 143, and the client received an empty reply, which curl reports as error 52. In a real service, nobody would know whether the card was charged, because the process died between sending the payment request and recording the result.

Graceful Shutdown

Graceful shutdown is stopping a service in an order that loses no work: stop accepting new work, finish the work already started, then exit. In Checkout, the SIGTERM handler only starts the shutdown, and the main thread does the rest after its serve loop returns:

def on_sigterm(signum, frame): log("SIGTERM received: accepting no new orders, finishing open ones") threading.Thread(target=server.shutdown).start() # ... server.serve_forever() server.server_close() log("all open orders finished, exiting with code 0")

server.shutdown() stops the serve loop, and server.server_close() closes the listening socket and then waits for every open request to finish. The graceful test ran the same steps with the handler installed, and it also sent SIGHUP before the order. It printed this log:

Output:

0.00s listening on port 8080 0.97s SIGHUP received: config reloaded, payment_timeout_ms=5000 1.28s order received, calling the payment provider (timeout 5000 ms) 1.78s SIGTERM received: accepting no new orders, finishing open ones 3.28s order 1042 placed, reply sent 3.28s all open orders finished, exiting with code 0 $ echo $? (exit code of Checkout) 0 $ curl output {"order_id": 1042, "item": "coffee grinder", "status": "placed"} (HTTP 201)

The signal arrived at 1.78 s, while the order was waiting for the payment provider. Checkout stopped accepting connections, let the order finish at 3.28 s, and exited with code 0. The client received its order number.

Two ways to stop
Two ways to stop

A real service follows the same order with more steps. A load balancer usually needs a few seconds to stop sending new requests after the service begins to stop, and connections to the database should close only after the last request that uses them is done.

The Grace Period and SIGKILL

A deploy tool or service manager cannot wait forever for a process to stop. It sends SIGTERM, waits for a fixed time called the grace period, and then sends SIGKILL. The grace period is 90 seconds by default in systemd, 10 seconds for docker stop, and 30 seconds in Kubernetes. Another test ran a process that ignores SIGTERM:

Output:

$ kill -TERM <pid>; sleep 3; ps -o pid,stat,comm -p <pid> PID STAT COMMAND 2681 S python3 $ kill -KILL <pid>; wait <pid>; echo $? 137

Three seconds after SIGTERM, the process was still running, with state S (sleeping). SIGKILL then ended it, and the shell reported 137. A service whose open requests can take longer than the grace period loses them in exactly this way, so request timeouts and the grace period must be chosen together.

SIGHUP and Reloading

SIGHUP originally told a program that its terminal had disconnected. Servers have no terminal, so many of them use it as a request to reload their settings without stopping. Nginx reloads its configuration on SIGHUP, and Gunicorn reloads its configuration and replaces its workers.

In the graceful test, Checkout's settings file was changed and the process received SIGHUP before the order arrived. The log line at 0.97 s shows the reload, and the order that followed used the new timeout of 5,000 ms instead of the old 2,000 ms. The process kept running through the reload, with the same PID and the same listening socket.

Exit Codes

An exit code is a number from 0 to 255 that a process returns to its parent when it ends; 0 means success, and any other value means failure. The program chooses the value, and the shell shows it as $?:

Output:

$ python3 -c "import sys; sys.exit(3)"; echo $? 3 $ python3 -c "raise ValueError(\"bad order\")"; echo $? ValueError: bad order 1

An uncaught Python exception produces exit code 1. When a signal ends a process, the process has no chance to choose, and shells report 128 plus the signal number. That gives 143 for SIGTERM (128 + 15) and 137 for SIGKILL (128 + 9). Exit code 137 in a crash report therefore means that something sent SIGKILL. The usual senders are a deploy tool after its grace period and the kernel when it runs out of memory, though a person running kill -9 gives the same code.

Zombies and Orphans

When a process ends, the kernel keeps its process ID and exit code until the parent collects them with the wait system call. A zombie is a process that has ended but whose parent has not yet called wait for it. The next test started a parent that never calls wait:

Output:

PID PPID STAT COMMAND 2689 2660 S python3 2691 2689 Z python3

The child, process 2691, has state Z. It uses no CPU, and its memory has been freed except for a small kernel record, but it keeps its process ID. A parent that creates many children and never collects them can fill the system's process table, and then no new process can start.

An orphan is a process whose parent has ended before it. The kernel gives every orphan a new parent, and that new parent calls wait for it later:

Output:

2695 1 S python3 PID COMMAND 1 systemd

On this machine, the new parent was PID 1, which is systemd. The new parent is PID 1 unless an ancestor of the orphan has asked the kernel to adopt orphans in its place; such a process is called a subreaper. This is one reason PID 1 has a special job on any Linux system: it collects the exit codes of orphans that nobody else will collect.

In Production

SymptomLikely causeHow to check
A few requests fail with empty replies during every deployNo SIGTERM handler, or the handler exits without finishing open requestsLook for exit code 143 in the service manager's log, such as systemctl status <service>
Exit code 137 after every deployThe service ignores SIGTERM or takes longer than the grace periodTime the shutdown and compare it with the grace period
Exit code 137 at random timesThe kernel killed the process for using too much memoryjournalctl -k | grep -i "killed process"
Many Z processes and "fork: Resource temporarily unavailable"A parent that never calls waitps -eo pid,ppid,stat,comm | grep ' Z'
A config change has no effectThe service does not reload on SIGHUP, or was never sent oneSend kill -HUP <pid> and read the log

Using This in an Interview

A common question is how a service should shut down during a deploy. A good answer is: "On SIGTERM, the service stops accepting new connections, finishes the requests already in progress, closes its connections, and exits with code 0, all within the grace period. After the grace period, the platform sends SIGKILL, which cannot be handled."

  • "Why not just handle SIGKILL too?" The kernel does not allow a handler for SIGKILL. It is the guaranteed way to stop a process, so a shutdown plan must finish before it arrives.
  • "What does exit code 137 tell you?" The process was ended by SIGKILL, because 137 is 128 plus 9. The usual sender is a deploy tool after the grace period or the kernel's out-of-memory handling.

Key Takeaways

  • A signal is a numbered notice from the kernel; a process can handle or ignore most signals, but never SIGKILL or SIGSTOP.
  • Without a SIGTERM handler, a Python service dies in the middle of a request; the demo exited with 143 and the client got an empty reply.
  • Graceful shutdown stops accepting work, finishes open work, and exits; with a handler, the open order finished and the process exited with 0.
  • Platforms send SIGTERM, wait for a grace period, and then send SIGKILL, which produced exit code 137.
  • Shells report 128 plus the signal number for a process ended by a signal.
  • A zombie holds a process ID until its parent calls wait, and an orphan is adopted, usually by PID 1, which collects it.

Signals, process IDs, and PID 1 behave a little differently when a service runs in a container, because the container gives it its own view of the system. The next lesson, A Container Is Still a Process, shows exactly what that view changes and what stays the same.

Practice Questions

Try each question first, then open the answer.

1. A Node.js service shows exit code 143 in its logs after every deploy, and a few requests fail each time. What happened, and what is the fix?

<details> <summary>Show answer</summary>

SIGTERM ended the service while requests were open, because it has no handler that finishes them. Exit code 143 is 128 plus 15, the number of SIGTERM. The fix is a handler that stops accepting new connections, waits for open requests to finish, and then exits with code 0.

</details>

2. A service's longest request can take 45 seconds, and it runs under docker stop with the default grace period. What can go wrong?

<details> <summary>Show answer</summary>

Requests still running after 10 seconds are cut off by SIGKILL. The default grace period for docker stop is 10 seconds. Either the grace period must be raised above the longest request time, or long requests must be limited by a shorter timeout.

</details>

3. ps shows 3,000 processes with state Z, all with the same parent. Will killing the zombies with SIGKILL remove them?

<details> <summary>Show answer</summary>

No. A zombie has already ended, so a signal has nothing to stop. It disappears only when its parent calls wait. Fixing or restarting the parent removes them: when the parent ends, the zombies become orphans, and PID 1 collects them.

</details>

4. Why does Checkout's SIGTERM handler start server.shutdown() in a new thread instead of calling it directly?

<details> <summary>Show answer</summary>

The handler runs in the main thread, which is the thread running serve_forever(). shutdown() waits for that loop to stop, so calling it from the same thread would wait for itself forever. Starting it in another thread lets the handler return, the loop notice the request, and the main thread continue to server_close().

</details>

5. A team wants to apply a new payment timeout without restarting Checkout. How does the demo do it, and what does the log show?

<details> <summary>Show answer</summary>

It changes the settings file and sends SIGHUP, and the handler reloads the file. The log shows "SIGHUP received: config reloaded, payment_timeout_ms=5000" at 0.97 s, and the order that arrived next used the 5,000 ms timeout. The process kept running through the reload, and its listening socket stayed open.

</details>

Reading Progress

0%


Vote for new content

On This Page

What a Signal Is

Stopping Without a Handler

Graceful Shutdown

The Grace Period and SIGKILL

SIGHUP and Reloading

Exit Codes

Zombies and Orphans

In Production

Using This in an Interview

Key Takeaways

Practice Questions