Solution for No space left on device error ~ inode depletion edition
I’ll introduce the solution when No space left on device error occurs on Linux servers despite having disk space available. The cause was inode depletion.
Disk usage was 76% with 3.7G free space, which shouldn’t cause No space left on device errors.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 16G 12G 3.7G 76% /
tmpfs 499M 0 499M 0% /dev/shm
I couldn’t figure out the cause and searched online to find:
While investigating various things, the theory that inodes were insufficient rather than capacity emerged.inode is the name of a file system used in UNIX-like OS, and it manages file owners, access permissions, size, creation date/time, location, etc.
Remaining inodes can be checked by adding -i to the df command, and when I tried it… bingo.
I found this article.
When I tried it immediately, bingo. It was an inode depletion problem.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda2 1048576 1048576 0 100% /
tmpfs 127541 1 127540 1% /dev/shm
I was able to identify directories that consume many inodes with the following script.
echo "==== `pwd` ====" ; for i in `ls -1`;do echo -n "## $i ### " ; echo "(`find ./$i -type f |wc -l`)" ;done | sort -r
- Source: inodeとは/inode消費が多いディレクトリの確認
From the root directory, I repeatedly executed the above script and used cd to move to lower hierarchy directories to identify the problematic directory.
When I encountered the No space left on device error, the cause was nearly 880,000 cache files from the WordPress plugin wp-file-cache.
I removed the wp-file-cache plugin as described in the following article.
After deleting files and checking IUse% with df -i, it dropped from 100% to 15%. I successfully resolved the inode depletion problem, and No space left on device errors stopped occurring.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda2 1048576 153319 895257 15% /
tmpfs 127541 1 127540 1% /dev/shm
That’s all from the Gemba where I identified inode depletion from No space left on device errors and successfully resolved it.