0% completed
Files, Inodes, and Disk Space
On This Page
Inodes
Hard and Symbolic Links
Deleted Files That Still Use Space
Inode Exhaustion
The Open Files Limit
Permissions
Mounts
Using This in an Interview
Key Takeaways
Practice Questions
An alert reports that the /var file system on a server is full, and a web application can no longer write its logs. The two standard tools disagree about the cause.
df -h /var Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 100G 100G 0 100% /var du -sh /var 40G /var
The df command reports 100 GB in use, but du, which adds up the files it can find, reaches only 40 GB. Neither tool is wrong. The missing 60 GB belongs to a file that has been deleted but is still open.
To explain that gap, the sections below follow how Linux stores a file, how it releases the space, and which other limits can stop writes.
Inodes
An inode is the record that stores everything about a file except its name: its type, owner, group, permissions, size, timestamps, link count, and the location of its data blocks. A directory entry maps a name to an inode number, and the data itself sits in blocks, fixed-size units of disk space. The ls -li command shows the inode number first.
ls -li /var/log/app 262147 -rw-r----- 1 app app 52428800 Nov 18 09:14 access.log
Here, inode 262147 has 1 link, belongs to user app and group app, and holds 52,428,800 bytes, which is 50 MiB. Because the name is stored in the directory and not in the inode, renaming a file inside one file system changes only the directory entry.
Hard and Symbolic Links
A hard link is a second directory entry for the same inode, and a symbolic link is a small separate file that contains a path. Running ln access.log copy.log raises the link count of inode 262147 to 2, and both names lead to the same data. Running ln -s access.log current.log creates a new inode whose content is the text "access.log".
| Hard link | Symbolic link | |
|---|---|---|
| Points to | An inode | A path |
| Across file systems | No | Yes |
| If the original name is deleted | Data remains through the other name | The link points to nothing |
| Raises the link count | Yes | No |
Deleted Files That Still Use Space
The kernel frees a file's blocks only when its link count is 0 and no process still has the file open. Deleting a file with rm removes a directory entry and lowers the link count. A process that opened the file earlier still holds a file descriptor, a small number that the process uses to refer to an open file, so the data remains.
That is the 60 GB gap. A log rotation job deleted app.log, but the application kept writing through its file descriptor. The du command reads directory entries and cannot see the file, while df counts used blocks and does. The lsof +L1 command lists open files whose link count is below 1.
lsof +L1 /var COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME java 3307 app 7w REG 259,1 64424509440 0 262151 /var/log/app/app.log (deleted)
The size, 64,424,509,440 bytes, is exactly 60 GiB, and FD 7w means descriptor 7, open for writing. Restarting the process frees the space. If a restart is not yet safe, truncate -s 0 /proc/3307/fd/7 empties the file through the descriptor. The lasting fix is to configure log rotation so that the application reopens its log, or so that rotation copies and empties the file in place.
Inode Exhaustion
A file system can run out of inodes while it still has free blocks, and then writes fail with "No space left on device." On ext4, the number of inodes is fixed when the file system is created. A common setting creates one inode for every 16 KiB, so a 100 GiB file system receives 107,374,182,400 / 16,384, or 6,553,600, inodes.
Suppose that an application stores sessions as 1 KB files. Each file still needs one 4 KiB block, so 6,553,600 files use every inode but only about 25 GiB of blocks. The df -i command shows the problem, which df -h does not.
df -i /srv Filesystem Inodes IUsed IFree IUse% Mounted on /dev/nvme2n1 6553600 6553600 0 100% /srv
Deleting old small files, or moving them into a database, frees inodes. XFS allocates inodes as it needs them, so it rarely has this problem.
The Open Files Limit
Each process may hold only a limited number of file descriptors, and network connections and pipes count toward that limit as well as files. A pipe is a channel that carries data from one process to another. The limit is set per process, and many systems use a default soft limit of 1024. A process that reaches it receives the error "Too many open files."
cat /proc/3307/limits | grep "open files" Max open files 1024 524288 files
The first number is the soft limit that applies now, and the second is the hard limit to which the process may raise it. The ulimit -n command shows the soft limit for the current shell, and a systemd service sets its own limit with LimitNOFILE=. Counting the entries in /proc/3307/fd shows how many descriptors the process holds.
Permissions
Every inode has an owner, a group, and three sets of permissions: for the owner, for the group, and for all other users. Each set contains r (read), w (write), and x (execute). In -rw-r-----, the owner can read and write, the group can read, and others have no access, which is mode 640 in octal (6 = 4 + 2, 4, and 0).
On a directory, r allows listing the names, w allows creating and deleting entries, and x allows entering the directory. Deleting a file therefore requires write permission on the directory, not on the file.
Mounts
A mount attaches a file system to a directory, and each mounted file system has its own blocks and inodes. The findmnt command lists them, and df reports each one separately, so a full /var does not mean that / is full.
A mount can also hide files. If a process wrote 20 GB into /data before a disk was mounted there, those files still use space on the parent file system. Neither du nor ls can see them until the mount is removed, which is a second cause of a gap between df and du.
Using This in an Interview
"df says the disk is full, but du finds much less. Why?" is a common systems question. Name both causes and the commands that separate them.
"df counts used blocks, while du adds up the files it can reach through directories. The usual cause of a gap is a deleted file that a process still holds open, which lsof +L1 shows. I would restart or signal that process, or truncate the file through /proc, and then fix log rotation. Less often, files are hidden under a mount point."
Expect two follow-up questions:
- "df -h shows free space, but writes fail. What next?" Run
df -i, because the file system may have run out of inodes. - "Why not delete the large log file to free space?" If a process still holds it open, deleting it frees nothing; truncating it does.
Key Takeaways
- An inode stores a file's details, such as owner, size, and data locations; the name lives in a directory entry.
- Space is freed only when no name and no open file descriptor remains.
lsof +L1finds deleted files that still use space, anddf -ifinds inode exhaustion.- File descriptors cover files, network connections, and pipes, and each process has a limit.
- Deleting a file needs write permission on its directory.
The next lesson, How DNS Resolves a Name, moves from the local machine to the network, starting with the lookup behind almost every connection.
Practice Questions
1. A teammate deletes a 30 GB log file, but df still shows the same usage. What is the likely cause, and how do you free the space?
<details><summary>Show answer</summary>A process still holds the file open, so its blocks remain allocated. Deleting the name lowered the link count to 0, but the open file descriptor still refers to the inode. Find the process with lsof +L1, then restart it or truncate the file through its descriptor in /proc.
2. A file system has 2,000,000 inodes and 500,000 free blocks, and writes fail. df -i shows 100 percent. What happened?
The file system ran out of inodes, so it cannot create new files. Free blocks do not help, because each new file also needs an inode. Find the directories with the most small files, and delete those files or combine them into fewer files.
</details>3. A user cannot delete a file with mode 666 in a directory with mode 755 owned by root. Why?
<details><summary>Show answer</summary>Deleting a file changes the directory, and only root can write to that directory. Mode 755 gives write permission only to the owner, root. The file's own mode, 666, controls its contents, not the removal of its name.
</details>Reading Progress
0%
On This Page
Inodes
Hard and Symbolic Links
Deleted Files That Still Use Space
Inode Exhaustion
The Open Files Limit
Permissions
Mounts
Using This in an Interview
Key Takeaways
Practice Questions