0% completed
What a Process Owns
On This Page
The Process and /proc
The Program and Where It Runs
Memory: the Address Space
Open Files: File Descriptors
Settings, Limits, and Identity
In Production
Key Takeaways
Practice Questions
A security review of Checkout found a problem that needed no exploit at all. Any program running under the same Linux user as Checkout could print its payment API key with a single command, and so could anyone with root access. The key was never written to a file or a log. It was stored in Checkout's environment, and the kernel shows a process's environment through a file.
That file is one of dozens that Linux keeps for every running process. Together they list everything the process owns: its memory, its open files and connections, its settings, and the identity it runs under.
This page opens a running Checkout process through those files and explains each thing it owns, so that you can inspect any process on a server the same way.
The Process and /proc
A process ID (PID) is the number that the kernel gives each process when it starts, and that number names the process in every tool and system call. Every process also has a parent process ID (PPID), which is the PID of the process that started it.
/proc is a virtual file system: its files are not stored on any disk, and the kernel builds their contents at the moment a program reads them. Each process has a folder there named after its PID, so /proc/2745 describes the process with PID 2745. Tools such as ps, top, and lsof get most of their information by reading these files.
The demo started a version of Checkout that holds what a real service holds: a log file, a connection to a database on port 5432, a listening socket on port 8080, and 50 MB of product data in memory. It also received two settings through environment variables. It ran on a cloud virtual machine with 2 CPUs and Linux kernel 7.0, and these are the first 24 entries of its folder:
Output:
arch_status attr autogroup auxv cgroup clear_refs cmdline comm coredump_filter cpu_resctrl_groups cpuset cwd environ exe fd fdinfo gid_map io ksm_merging_pages ksm_stat latency limits loginuid map_files
The Program and Where It Runs
Three entries say which program the process is and where it is working. cmdline holds the command line that started it, exe is a link to the program file, and cwd is a link to the current working directory, which is the folder that relative file names start from:
Output:
python3 checkout.py /usr/bin/python3.12 /tmp/checkout
The program is the Python interpreter, /usr/bin/python3.12, and checkout.py is only an argument to it. Checkout changed its working directory to /tmp/checkout when it started, so a relative name such as orders.log would mean /tmp/checkout/orders.log.
The status file summarizes the process. These are nine of its lines:
Output:
Name: python3 State: S (sleeping) Pid: 2745 PPid: 2739 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 VmSize: 80812 kB VmRSS: 70980 kB Threads: 1
State: S (sleeping) means the process is blocked, waiting for a connection, and is using no CPU. Threads: 1 means one thread runs the program, which is the case for Python's simple server.
Memory: the Address Space
An address space is the range of memory addresses that a process can use; the kernel maps each part of it to real memory or to a file, and no other process can see it. Two processes can use the same address and see completely different data, because each address space has its own map.
VmSize in the status lines is the size of the whole address space, about 79 MB. VmRSS is the resident set size (RSS), which is the part that is in physical memory right now, about 69 MB. Most of that RSS is the 50 MB of product data, because Checkout wrote to every page of it:
cache = bytearray(b"\x01") * (50 * 1024 * 1024) # 50 MB of product data kept in memory
The maps file lists the address space one region at a time. Each line shows an address range, its permissions (r read, w write, x execute, p private), and what the region holds:
Output:
00400000-00420000 r--p 00000000 08:01 2110 /usr/bin/python3.12 00420000-00703000 r-xp 00020000 08:01 2110 /usr/bin/python3.12 00703000-00a28000 r--p 00303000 08:01 2110 /usr/bin/python3.12 00a28000-00a29000 r--p 00627000 08:01 2110 /usr/bin/python3.12 00a29000-00ba7000 rw-p 00628000 08:01 2110 /usr/bin/python3.12 14122000-14371000 rw-p 00000000 00:00 0 [heap]
The first five lines are the Python program file, mapped into memory in parts. Only one part is executable (r-xp), and the code cannot be written to. The last line is the heap, which is the region where the program's ordinary memory allocations live.
Open Files: File Descriptors
A file descriptor is a small number that a process uses to name an open file, socket, pipe, or device; the kernel keeps the real object in a table that belongs to the process. Every read, write, accept, and close call takes a file descriptor as its first argument. The fd folder in /proc holds one link per open descriptor:
Output:
0 -> /dev/null 1 -> /tmp/checkout/stdout.log 142 -> pipe:[13115] 145 -> pipe:[13116] 2 -> /tmp/checkout/stdout.log 3 -> /tmp/checkout/orders.log 4 -> socket:[19523] 5 -> socket:[20753]
Descriptors 0, 1, and 2 are standard input, standard output, and standard error. Every process starts with them, and here they point to /dev/null and to a log file. Descriptor 3 is orders.log, which Checkout opened first. Descriptor 4 is the connection to the database, and descriptor 5 is the listening socket, because the kernel always gives out the lowest free number.
A socket appears as socket:[19523], where 19523 is the socket's inode number, a number that identifies the kernel object. ss -tanpe printed the same numbers: ino:19523 for the connection from port 43940 to port 5432, and ino:20753 for the socket listening on port 8080.
Descriptors 142 and 145 are a surprise, because Checkout never opened them. They are pipes that the program which started the demo had open, and Checkout received them when it was started. A program started by another process receives copies of that process's open descriptors, except those marked close-on-exec. Close-on-exec is a flag that tells the kernel to close the descriptor when the process switches to running a new program. A leaked descriptor like these can keep a pipe, a file, or even a listening port open long after its owner expects it to be closed.
Settings, Limits, and Identity
The environment is a list of NAME=value settings that a process receives from its parent when it starts. The environ file holds it, separated by zero bytes:
Output:
PAYMENT_API_KEY=demo-secret-123 CHECKOUT_DB_URL=postgres://checkout@127.0.0.1:5432/orders HOME=/home/runner USER=runner
Two of these settings are Checkout's own, and PAYMENT_API_KEY is the one the security review found. The limits file shows the resource limits, which are ceilings that the kernel enforces for the process:
Output:
Limit Soft Limit Hard Limit Units Max stack size 16777216 unlimited bytes Max processes 28076 28076 processes Max open files 65536 65536 files
Checkout may hold at most 65,536 descriptors at once, counting files and sockets together, and its user may run at most 28,076 processes. The soft limit is the one in force, and a process may raise it up to the hard limit.
Credentials are the user and group IDs that a process runs as, and the kernel checks them on every access to a file or another process. The line Uid: 1001 1001 1001 1001 lists four user IDs: the real, effective, saved, and file system IDs. They are all 1001, the user runner. Another user cannot read Checkout's environment:
Output:
cat: /proc/<pid>/environ: Permission denied
The same check protects the other sensitive entries, such as maps, mem, and the fd folder: only root and processes running as user 1001 can read them. Other entries, such as status, limits, and cmdline, can be read by every user on the machine, so a secret passed as a command-line argument is visible to everyone.
In Production
Each thing a process owns has a matching problem in production, and /proc shows each one without restarting the process.
| Symptom | Where it shows | Command |
|---|---|---|
| A secret leaks to anyone who shares the service's user | environ | tr '\0' '\n' < /proc/<pid>/environ |
| "Too many open files" errors | The number of links in fd against Max open files | ls /proc/<pid>/fd | wc -l, grep 'open files' /proc/<pid>/limits |
| A port stays busy after the service stops | A descriptor inherited by a child process | ls -l /proc/*/fd 2>/dev/null | grep socket, or ss -tlnp |
| The service reads the wrong config or writes logs to an odd place | cwd and relative paths | readlink /proc/<pid>/cwd |
| Memory grows over days | VmRSS in status | grep VmRSS /proc/<pid>/status |
| The service cannot read a file that you can read | Credentials differ from yours | grep -E 'Uid|Gid' /proc/<pid>/status |
Two habits follow from this. First, keep secrets out of the environment where you can, or at least know who can read it. Second, check limits on the running process, not in your shell, because a service started by a service manager often has different limits from your login session.
Key Takeaways
/proc/<pid>is built by the kernel on demand and shows everything a process owns, without stopping it.- The address space is the process's private memory map;
VmSizeis its full size andVmRSSis the part in physical memory. - A file descriptor is a small number in a per-process table; the kernel gives out the lowest free number, and sockets appear as
socket:[inode]. - A started program receives its parent's environment and its open descriptors, except descriptors marked close-on-exec.
- The environment can be read by the same user and by root, while entries such as
cmdlineandstatuscan be read by every user. - Credentials and resource limits belong to the process, so check them in
/proc/<pid>, not in your shell.
Checkout received its descriptors and its environment from the process that started it. The next lesson, Fork, Exec and How Servers Start Workers, shows how that copying happens, and how servers such as Gunicorn use it to start many workers that share one listening socket.
Practice Questions
Try each question first, then open the answer.
1. Checkout opened orders.log, then a database connection, then a listening socket. Why did the listening socket get descriptor 5?
The kernel always gives out the lowest free descriptor number. Descriptors 0, 1, and 2 were already taken by the standard streams, so the log file got 3, the database connection got 4, and the listening socket got 5. The inherited descriptors 142 and 145 did not change this, because lower numbers were free.
</details>2. VmSize is about 79 MB and VmRSS is about 69 MB. What is the difference between the two numbers?
VmSize is the size of the whole address space, and VmRSS is the part that is in physical memory right now. Address space can be reserved without being used. Memory becomes resident only when the process touches it, as Checkout did when it filled its 50 MB of product data.
3. A service runs fine when you start it from your shell, but under the service manager it fails with "Too many open files" at a much lower load. What would you compare?
<details> <summary>Show answer</summary>Compare Max open files in /proc/<pid>/limits of the running service with ulimit -n in your shell. Limits belong to each process and are inherited from whatever started it. A service manager often starts services with a lower limit than a login shell has.
4. You stop an old server process, but its port stays busy, and ss -tlnp shows a different program holding the listening socket. How could that happen?
The old server started a child process, and the child inherited the listening socket's descriptor. The socket stays open while any process holds a descriptor for it. Marking descriptors close-on-exec, which most modern runtimes do by default, prevents a started program from inheriting them.
</details>5. A teammate argues that PAYMENT_API_KEY in the environment is safe because the variable never appears in any log. Is that right?
No. The kernel shows the environment in /proc/<pid>/environ to root and to every process of the same user. The demo printed the key with one command. Child processes also receive a copy of the environment when they start, so the key spreads to every program that Checkout runs.
Reading Progress
0%
On This Page
The Process and /proc
The Program and Where It Runs
Memory: the Address Space
Open Files: File Descriptors
Settings, Limits, and Identity
In Production
Key Takeaways
Practice Questions