Grokking DevOps and SRE Interviews
Vote

0% completed

​

Memory, CPU, and Load

Virtual and Resident Memory

Page Cache and Available Memory

Swap

The OOM Killer

CPU Time

Load Average

Cgroup Limits

Using This in an Interview

Key Takeaways

Practice Questions

Title image: a 16 GB server with only 410 MiB free but about 11,420 MiB available, because most memory holds page cache.
Title image: a 16 GB server with only 410 MiB free but about 11,420 MiB available, because most memory holds page cache.

A monitoring alert reports that a server with 16 GB of memory has only 410 MiB free, about 2.6 percent of the total. The same server shows a load average of 6.1 on 4 CPUs, while its CPUs are 30 percent idle.

Both numbers look alarming, but neither means what it appears to mean. Most used memory holds file data that the kernel can release at once, and the load includes processes waiting for a disk.

What follows explains how Linux accounts for memory and CPU time, so that you can read these numbers.

Virtual and Resident Memory

Virtual memory is the address space that a process has reserved, while resident memory is the part that currently occupies physical RAM. The top command shows them as VIRT and RES, and ps shows them as VSZ and RSS. A Java service might show 8.2 GB virtual and 1.1 GB resident, because it has used only part of the range it reserved.

Resident memory is the figure that matters for RAM. A sum of RSS values across processes still overstates the total, because a shared library used by 20 processes is counted 20 times.

Page Cache and Available Memory

The page cache is file data that the kernel keeps in unused memory, so that the next read comes from RAM instead of disk. When a program needs memory, the kernel releases cache pages first. Unused RAM gives no benefit, so a healthy server shows little free memory.

free -m total used free shared buff/cache available Mem: 15990 4210 410 95 11370 11420 Swap: 4095 0 4095

Read the available column, which is the kernel's estimate of memory that new work can use without swapping. Only 410 MiB is free, but about 11,420 MiB is available, because most of the 11,370 MiB of cache can be released. Available is below free plus cache, 11,780 MiB, because some cached pages are in use or not yet written to disk.

Where memory goes on a 16 GB server: 4,210 MiB used by processes, 11,370 MiB of buffers and page cache, and 410 MiB free, so about 11,420 MiB is available.
Where memory goes on a 16 GB server: 4,210 MiB used by processes, 11,370 MiB of buffers and page cache, and 410 MiB free, so about 11,420 MiB is available.

An alert on free memory would trigger on this healthy server, but an alert on available memory would not.

Swap

Swap is disk space to which the kernel moves memory pages that have not been used recently. A disk is far slower than RAM, so a process slows down when it needs a page that was swapped out. A little swap in use is normal. Continuous movement is the warning sign, shown in the si (swap in) and so (swap out) columns of vmstat.

The OOM Killer

When the kernel can neither release cache nor swap enough memory, the out-of-memory (OOM) killer ends a process to recover RAM. It scores each process mainly by its memory use and sends SIGKILL to the highest score. An OOM kill therefore appears as exit code 137, as How Linux Runs a Process explained.

An administrator can change the choice with oom_score_adj, a value from -1000 to 1000 for each process, where -1000 exempts the process. The kernel log records every kill.

dmesg | grep -i "killed process" [3542117.204118] Out of memory: Killed process 4121 (java) total-vm:9437184kB, anon-rss:6291456kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:13312kB oom_score_adj:0

The anon-rss value, 6 GiB, is the memory that the program allocated for itself, such as its heap, and that is held in RAM. The total-vm value, 9 GiB, is the virtual size.

CPU Time

Linux divides each CPU's time into categories, and the category, not the total, shows the kind of problem. The top command prints the split for all CPUs in its header.

top - 10:42:17 up 41 days, 3:12, 1 user, load average: 6.12, 5.80, 4.95 Tasks: 213 total, 1 running, 212 sleeping, 0 stopped, 0 zombie %Cpu(s): 22.1 us, 4.3 sy, 0.0 ni, 30.2 id, 42.6 wa, 0.0 hi, 0.4 si, 0.4 st MiB Mem : 15990.0 total, 410.0 free, 4210.0 used, 11370.0 buff/cache MiB Swap: 4095.0 total, 4095.0 free, 0.0 used. 11420.0 avail Mem
FieldMeaningA high value suggests
us (user)Running application codeBusy application code
sy (system)Running kernel code for processesMany system calls (requests to the kernel), such as small reads or network work
id (idle)Nothing to runSpare capacity
wa (iowait)Idle while at least one disk request is outstandingSlow storage, while the CPU itself is free
st (steal)A virtual CPU was ready, but the hypervisor ran another machineOther machines competing for the same cloud host

A hypervisor is the software that runs virtual machines on a physical host. Here, 42.6 percent iowait with 30.2 percent idle means that processes are waiting for the disk, so a faster CPU would not help.

Load Average

The load average is the average number of processes that are running, waiting for a CPU, or in uninterruptible sleep, over the last 1, 5, and 15 minutes. Linux counts processes in D state, which many other Unix systems do not. Waits for disk or network storage therefore raise the load even when the CPUs are idle.

Compare the load with the number of CPUs. If no process is in D state, a load of 4 on 4 CPUs means that every CPU was busy. A load of 8 means that about 4 processes were also waiting. The vmstat output separates the two causes.

vmstat 5 2 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 431104 81920 11549696 0 0 310 180 950 1600 9 2 87 2 0 1 5 0 419840 81920 11560960 0 0 9840 210 2310 4120 22 4 30 43 0

The first data line averages the time since boot, so read the second. Its r column shows 1 runnable process, and its b column shows 5 blocked in D state, which explains a load near 6. One busy process on 4 CPUs uses about 25 percent of CPU time, matching 22 percent user and 4 percent system. The rest of the load comes from the disk, which reads about 9,840 blocks a second.

A load of 6 on 4 CPUs: 1 process runs on a CPU, 3 CPUs wait idle, and 5 processes wait for the disk in D state.
A load of 6 on 4 CPUs: 1 process runs on a CPU, 3 CPUs wait idle, and 5 processes wait for the disk in D state.

Cgroup Limits

A control group (cgroup) is a kernel feature that places processes in a group and limits and measures their CPU, memory, and I/O. Container limits are built on cgroups: the runtime creates a group for each container and writes its limits into files. With cgroup version 2, you can read them from inside the container.

cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max
536870912 50000 100000

The memory limit is 536,870,912 bytes, or 512 MiB. If the group exceeds it and the kernel cannot release enough memory, the OOM killer acts inside the group alone, and the log says "Memory cgroup out of memory." The host may still have gigabytes free. The group's page cache counts toward its usage, but the kernel releases that cache before it ends any process.

The CPU line allows 50,000 microseconds of CPU time in every 100,000-microsecond period, or half of one CPU. A group that uses its quota early in a period must wait for the next period. This wait is called throttling, and it slows the process without ending it. How Kubernetes Schedules Pods shows how Kubernetes sets these values.

Using This in an Interview

"The free command shows almost no memory left. Is the server in trouble?" checks whether you read Linux memory correctly. Start with the column you trust, then name the real signs of pressure.

"Not necessarily. Linux uses spare RAM as page cache and releases it when programs need memory, so I read the available column instead of free. Real pressure shows as low available memory, swap-out activity in vmstat, or OOM kills in the kernel log. In a container, I also compare usage with the cgroup's memory.max."

Expect two follow-up questions:

  • "The load is 12 on an 8-CPU server, but CPU use is 40 percent. How?" Linux counts processes in D state, so waits for storage raise the load. Check the b and wa columns in vmstat, then the disks.
  • "Why was this process killed when the server had free memory?" It probably exceeded its cgroup memory limit, which the kernel enforces for each group separately.

Key Takeaways

  • Resident memory, not virtual memory, shows how much RAM a process holds.
  • Low free memory is normal; low available memory is the warning sign.
  • The OOM killer sends SIGKILL, so its victims exit with code 137.
  • High iowait points to storage, and high steal points to a busy cloud host, not to application code.
  • Container limits are cgroup limits: a group over its memory limit loses a process to the OOM killer, and a group over its CPU quota is throttled.

The next lesson, Files, Inodes, and Disk Space, turns from memory to storage and explains why a disk can be full when its files appear small.

Practice Questions

1. A 16 GB server shows 300 MiB free and 9,800 MiB available, with no swap activity. Does it need more memory?

<details><summary>Show answer</summary>

No, because most of its memory is page cache that the kernel can release. About 9,800 MiB is available for new work.

</details>

2. A server with 2 CPUs has a load average of 6, user time of 95 percent, and no processes in D state. What does the load tell you?

<details><summary>Show answer</summary>

On average, 2 processes were running and about 4 were waiting for a CPU. The load is three times the number of CPUs, and the high user time shows that application code uses them.

</details>

3. A container's cpu.max is "25000 100000." A request needs 100 ms of CPU time. About how long does it take to finish?

<details><summary>Show answer</summary>

At least about 325 ms, even on an idle host. The group may use 25 ms of CPU time in each 100 ms period. The request runs 25 ms and waits 75 ms three times, then runs its last 25 ms, so it ends after 3 x 100 + 25, or 325 ms.

</details>

Reading Progress

0%


Vote for new content

On This Page

Virtual and Resident Memory

Page Cache and Available Memory

Swap

The OOM Killer

CPU Time

Load Average

Cgroup Limits

Using This in an Interview

Key Takeaways

Practice Questions