What to do when your Linux file system shows its full, but really isn’t

I ran across an uncommon scenario whereby a file system on a Linux server was showing it was nearly full, but I could only account for about half of the space usage.

[root@web18 /]# df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/sda2              165G  135G   21G  87% /

After some research, it was determined that a stray process had a lock in place on files that were previously deleted. Since this lock was in place, the file system would not show the space as available.

[root@web18 /]# ls -l /proc/*/fd/* | grep deleted 
l-wx------ 1 nginx nginx 64 Aug  1 09:39 /proc/55627/fd/15 ->
 /var/log/nginx/access_log-20130731 (deleted)
l-wx------ 1 nginx nginx 64 Aug  1 09:39 /proc/55627/fd/16 ->
 /var/log/nginx/error_log-20130731 (deleted)

After killing the stale nginx process (killall -9 nginx), the space started showing its normal consumption.

[root@web18 /]# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       165G   67G   90G  43% /

I hope this helps someone who suffers from a similar problem

-James

About james

Chief Technology Officer Atlantic Metro Communications
This entry was posted in How-to and tagged , . Bookmark the permalink.