Ever heard of inodes? You need lots of them.

Ran into a situation on a customer’s CentOS server the other day where a service wasn’t working. Symptoms and error messages indicated the disk was full. However ‘$ df -h’ was showing ample free space. What the heck? Turned out the maximum number of files on the disk had been consumed. Technically speaking, the limiting factor was the number of inodes allocated to the volume. An inode is taken up for each file, directory and link on the file system. Inodes act like a database for the files on a file system and contain pointers to the actual information.

When a partition is created the maximum number of inodes is established, rather set in stone. There is no way to re-partition the number of inodes on the fly. In this particular case the volume was 75GB with 23GB free, but only 1,000,000 inodes were allocated to it. The temporary solution was to remove old files that were not needed to get the total number of files on the partition safely back below 1M. As soon as that was taken care of the system started working again.

Unix/Linux (and Mac of course) have the inode concept built into their file systems. To check out the inode status run ‘$ df -i’ to make sure you are not at risk of running out of those precious inodes.

user@host.com [~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda             49152000 8771724 40380276   18% /

inode related commands:

‘$ ls -i’ it will output the inode ids for each file / directory.

user@host.com [~]# ls -i1
 1725516 access-logs@
 1721190 backups/
 1720340 dead.letter
 1720652 etc/
 2173459 logs/
 1720654 mail/
 1720648 public_html/
41845314 python@
 1729306 ssl/
 1720653 tmp/
 1720660 www@

The stat command will tell more details about the particular file / inode.

user@host.com [~]# stat public_html
  File: `public_html'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 800h/2048d      Inode: 1720648     Links: 13
Access: (0750/drwxr-x---)  Uid: ( 1058/user)   Gid: (   99/  nobody)
Access: 2011-12-04 16:29:56.000000000 -0500
Modify: 2014-04-20 03:19:04.000000000 -0400
Change: 2014-05-17 00:00:11.000000000 -0400

To get a count of the inodes per folder under the current directory:

user@host.com [~]# find . -type f -printf "%h\n" | cut -d/ -f-2 | sort | uniq -c | sort -rn
   5789 ./public_html
    557 ./mail
    555 ./tmp
    205 ./logs
     75 ./.cpanel
     43 ./etc
     25 .
     13 ./.sqmaildata
     10 ./.fontconfig
      6 ./.subversion
      6 ./.gnupg
      6 ./.fantasticodata
      5 ./.htpasswds
      3 ./backups
      2 ./.emacs.d
      1 ./.ssh
      1 ./public_ftp
      1 ./.cpan

This can take forever so you may want to direct the output to a file (assuming you can spare an inode):

user@host.com [~]# find . -type f -printf "%h\n" | cut -d/ -f-2 | sort | uniq -c | sort -rn > inode_count.txt

For more information:

This entry was posted in Sys Admin and tagged , , . Bookmark the permalink.

Comments are closed.