No Space Left on Device

Issue: No Space Left on Device

Discussion:

I have a server which has been setup to capture uploaded crash logs. Every hour I transfer these files (using rsync) to another server for statistical analysis.  One of our end users noticed that the current day statistics were not as expected.  I manually ran the script and noticed the No space left on device error below.  At first didn't understand how I could be out of space when a df -h returned 14 Gigabytes free.  After a reboot, and a little Googling I realized that the system was out of available Inodes.  

For those new to Linux an Inode is simply a pointer to a file, or more precisely pointer or index to a block on a disk which contains either part of or all of a file. In the old days a writable block on a system might be 512 Kb, now it is typical to have a block size of 4096 Kb or even 8192 Kb.  If a file doesn't fit into a single block then the next available block is written to and the system "magically" keeps track of the next block. On my system each of these crash logs was very small, and over the previous 6 months we had over 1.6 million uploaded to the server each file took up one Inode, and eventually I ran out of Inodes even though I didn't run out of diskspace.

rsync: mkstemp "/usr/local/crashlogs/20141120/.20112014135212_0.log.MdO3DE" failed: No space left on device (28)

rsync: mkstemp "/usr/local/crashlogs/20141120/.20112014135217_0.log.qz7uz4" failed: No space left on device (28)

rsync: mkstemp "/usr/local/crashlogs/20141120/.20112014135219_0.log.73mgvu" failed: No space left on device (28)

#df -h

Filesystem                   Size  Used Avail Use% Mounted on

/dev/mapper/vg_tern-lv_root   26G   11G   14G  45% /

The out of disk space error was related to the lack of available Inodes.

#df -ih

Filesystem                  Inodes IUsed IFree IUse% Mounted on

/dev/mapper/vg_tern-lv_root   1.6M  1.6M     1  100% /

Solution:

  There are some versions of Linux, and file system types that will allow one to increase the number of available Inodes. Mine was not one of them.  One option would be to back everything up reformat the disk with more Inodes then restore the system.  This was not an option for me.  The final solution is to remove unnecessary files.  Lucky for me the end user didn't care about old data thus she gave me permission to remove three months worth of crash logs. This freed up over 500,000 inodes.

IF you don't know the directory that is causing you issues below is a one line script that can help you find the location of your many small files.

for i in /*;do echo $i; find $i | wc -l; done