Grokking Operating Systems and Networking for Backend Engineers
Vote

0% completed

​

Processes and Threads From the Kernel's View

Tasks and Thread Groups

One System Call, Different Flags

What Threads Share and What They Do Not

The Scheduler Sees Threads, Not Processes

In Production

Using This in an Interview

Key Takeaways

Practice Questions

Processes and Threads From the Kernel's View
Processes and Threads From the Kernel's View

A small Python version of Checkout starts four worker threads and then waits. ps lists it as one process with one process ID. The kernel, however, keeps a folder of tasks for every process, and for this one it printed five numbers:

Output:

2668 2670 2671 2672 2673

The first number is the process ID itself. The other four belong to the worker threads, and to the scheduler each of the five is a separate thing to run.

This page explains how Linux represents processes and threads, what threads share and what they keep for themselves, and what that means for a service that uses many threads.

Tasks and Thread Groups

A thread is one sequence of instructions running inside a process; a process can have many threads, and they all use the process's memory. Java and Go services run many threads even when the code does not create any directly, and many Python libraries start threads of their own.

Linux does not have two separate kinds of object for processes and threads. A task is the kernel's unit of scheduling: one sequence of instructions with its own registers, stack, and state, which the scheduler can place on a CPU. Every thread is a task, and a single-threaded process is a process with exactly one task.

Each task has its own ID, called the thread ID (TID). Tasks that belong to the same process form a thread group, and the thread group ID (TGID) is the number that tools call the process ID. The demo ran on a cloud virtual machine with 2 CPUs and Linux kernel 7.0, and ps -L printed one line per task:

Output:

PID TID NLWP STAT COMMAND 2668 2668 5 Sl python3 2668 2670 5 Sl python3 2668 2671 5 Sl python3 2668 2672 5 Sl python3 2668 2673 5 Sl python3

All five tasks share the PID 2668, and each has its own TID. NLWP is the number of tasks in the group, an old name that means "number of lightweight processes". The main thread's TID equals the PID. The status file of each task shows the same two numbers under the kernel's own names:

Output:

task 2668: Tgid: 2668 Pid: 2668 task 2670: Tgid: 2668 Pid: 2670 task 2671: Tgid: 2668 Pid: 2671 task 2672: Tgid: 2668 Pid: 2672 task 2673: Tgid: 2668 Pid: 2673 Threads: 5

Inside the kernel, Tgid is the process ID and Pid is the task's own ID, which is confusing but consistent. When you send a signal to a process, you send it to the thread group.

One System Call, Different Flags

Both new threads and new processes come from the same system call, clone. Its flags tell the kernel what the new task shares with the task that created it. The program share.py created one thread, then forked one child, while strace recorded the clone calls:

Output:

orders seen by the main thread: ['added by a thread'] files open in this process: ['0', '1', '142', '145', '2', '3', '4'] descriptor of the thread's file: 3 2703 clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=<addr>, parent_tid=<addr>, exit_signal=0, stack=<addr>, stack_size=0xffff80, tls=<addr>} => {parent_tid=[2704]}, 88) = 2704 2703 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=<addr>) = 2705

The first clone3 call made the thread. CLONE_VM shares the memory, CLONE_FILES shares the file descriptor table, CLONE_SIGHAND shares the signal handlers, and CLONE_THREAD puts the new task in the same thread group. The second call, clone with only SIGCHLD, made the forked child, which received copies instead, and SIGCHLD is the signal the parent gets when the child ends.

The first three lines of output show the effect of those flags:

def add_from_thread(): orders.append("added by a thread") global log log = open("/tmp/orders.log", "w") # a file opened by the thread

The thread's change to the list orders was visible to the main thread, but the forked child's change was not, because the child wrote to its own copy of the memory. The file that the thread opened appeared as descriptor 3 in the process's descriptor table, which every thread of the process uses.

What threads share, and what a forked child copies
What threads share, and what a forked child copies

What Threads Share and What They Do Not

Threads of one process share everything that belongs to the process:

  • Memory. Code, global variables, and the heap. A value written by one thread can be read by any other.
  • Open files and sockets. One descriptor table, so a socket accepted by one thread can be used by another.
  • Signal handlers, the current directory, and credentials.

Each thread keeps its own:

  • Stack. The memory that holds its function calls and local variables.
  • Registers and instruction position. The state that lets it run independently.
  • Thread ID and scheduling state. Whether it is running, ready to run, or sleeping.

Because memory is shared, two threads that change the same data at the same time can corrupt it. Coordinating them with locks and similar tools is outside this course; what matters here is that the kernel does nothing to prevent it.

The Scheduler Sees Threads, Not Processes

The scheduler places tasks on CPUs. A process with two busy threads can therefore use two CPUs at once. The demo machine has the following processors:

Output:

CPU(s): 2 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1

A C program ran the same fixed loop on one thread and then on two threads. The shell's time command reports real time, which is time on the clock, and user time, which is CPU time added over all threads:

Output:

1 thread(s) done real 0.414 s user 0.412 s sys 0.002 s

Output:

2 thread(s) done real 0.831 s user 1.648 s sys 0.001 s

With one thread, the loop took 0.41 s on the clock and 0.41 s of CPU time. With two threads, real time was 0.83 s and user time was 1.65 s, twice the real time, so both threads ran at the same moment, one on each CPU. Running at the same moment did not make the work finish sooner, though. Each thread took about twice as long as the single thread did alone, so two threads did no more work per second than one. The lscpu output shows one likely reason: the two CPUs are two hardware threads of one physical core (Thread(s) per core: 2), and two busy hardware threads share that core's circuits. On a cloud virtual machine, the host can also limit how much real processor time the two CPUs get, and this run cannot separate the two causes. A count of CPUs is therefore not a promise of that much computing power.

The same test in Python behaved differently:

Output:

1 thread(s) done real 0.865 s user 0.859 s sys 0.006 s

Output:

2 thread(s) done real 1.685 s user 1.683 s sys 0.005 s

One Python thread took 0.87 s. Two threads took 1.69 s on the clock and 1.68 s of CPU time, so user time no longer grew faster than the clock, and only one thread was running at any moment. The kernel was ready to run both. CPython's global interpreter lock (GIL), which lets only one thread run Python code at a time, made the other one wait. That limit belongs to the language runtime, not to the kernel, and a thread that is waiting for the network or a disk releases the lock.

Two threads of CPU work, in C and in Python
Two threads of CPU work, in C and in Python

In Production

SymptomWhat it meansCommand
A Python service uses at most about one CPU, however many threads it hasThe interpreter lock lets one thread run Python code at a timetop -H -p <pid> shows each thread's CPU
The thread count grows over hoursThreads are created and never finish, often one per stuck requestgrep Threads /proc/<pid>/status, ps -o nlwp -p <pid>
"Unable to create native thread" or "Resource temporarily unavailable"Threads count against the per-user process limit and the system's task limitgrep 'Max processes' /proc/<pid>/limits, cat /proc/sys/kernel/threads-max
One thread uses 100 percent of a CPU and the rest are idleA busy loop or a hot lock in one threadtop -H -p <pid>, then a stack dump of that thread

The thread ID in top -H is the same number as in /proc/<pid>/task, so a busy TID can be matched to a thread in a Java or Python stack dump that prints native thread IDs.

Using This in an Interview

The difference between a process and a thread is a very common question. A precise answer is: "To the Linux kernel, both are tasks created by clone. Threads of one process share memory, open files, and signal handlers, and each has its own stack and registers. A new process gets its own copy of those things, so a change in one process is invisible to the other."

  • "Why would you choose processes over threads?" Isolation. A crash or memory corruption in one process does not affect the others, and in Python, separate processes avoid the interpreter lock.
  • "Why choose threads?" Sharing is cheap. Threads read the same data without copying it, and creating a thread does not copy page tables.

Key Takeaways

  • Linux schedules tasks, and every thread is a task with its own thread ID.
  • The process ID is the thread group ID, and the main thread's TID equals it.
  • Threads and processes are both created by clone; the flags decide whether memory, descriptors, and signal handlers are shared or copied.
  • Threads share memory and the descriptor table, and each keeps its own stack, registers, and scheduling state.
  • Two C threads used 1.65 s of CPU time in 0.83 s on the clock, because the kernel ran them on both CPUs at once, although on this machine each ran at about half its solo speed.
  • Two Python threads used no more CPU time than clock time, because CPython's interpreter lock lets one thread run Python code at a time.

Because the kernel sends signals to the whole thread group, every thread of Checkout is affected when the deploy tool asks it to stop. The next lesson, Signals, Graceful Shutdown and Exit Codes, shows how a service should respond to that request.

Practice Questions

Try each question first, then open the answer.

1. top shows a Java service at 380 percent CPU on a machine with 4 CPUs. How can one process use more than 100 percent?

<details> <summary>Show answer</summary>

Its threads are separate tasks, and the scheduler can run four of them at once. top adds the CPU time of all of a process's threads, so 100 percent means one full CPU. top -H -p <pid> shows which threads use the time.

</details>

2. A thread in Checkout opens a connection to the database and stores the descriptor in a global variable. Can another thread of the same process send queries on it? Could a forked child do the same?

<details> <summary>Show answer</summary>

Yes for the thread, and yes for the child, but in different ways. The thread uses the same descriptor table, so the same number names the same connection. A forked child gets a copy of the table, so its descriptor also points at the same connection, and two processes then share one connection without knowing it. That usually mixes their queries and replies, which is why connections should be opened after forking.

</details>

3. ls /proc/<pid>/task for a Go service shows 14 entries, although the code never created a thread. Where did they come from?

<details> <summary>Show answer</summary>

The runtime created them. Go's runtime, the JVM, and Python's libraries start threads of their own, for example for garbage collection or for system calls that block. Every one of them is a task, so it appears in the task folder.

</details>

4. In the Python test, why was user time close to real time with two threads, while in C user time was about twice real time?

<details> <summary>Show answer</summary>

In Python, only one thread can run Python code at a time, so the CPU time cannot grow faster than the clock. Both threads were tasks that the kernel could schedule, but one of them was always waiting for the interpreter's lock. The C threads had no such lock, so both ran on the two CPUs at once and their CPU time added up.

</details>

5. You send SIGTERM to the PID of a process with five threads. Which thread receives it?

<details> <summary>Show answer</summary>

The signal goes to the whole thread group, and the kernel delivers it to one thread that has not blocked it. In Python, the handler always runs in the main thread. If there is no handler, the default action ends the whole process, all five tasks together.

</details>

Reading Progress

0%


Vote for new content

On This Page

Tasks and Thread Groups

One System Call, Different Flags

What Threads Share and What They Do Not

The Scheduler Sees Threads, Not Processes

In Production

Using This in an Interview

Key Takeaways

Practice Questions