rm not freeing diskspace

Question

Hi everyone,

I've rm'ed a 2.5gb log file - but it doesn't seemed to have freed any space.

I did:

rm /opt/tomcat/logs/catalina.out

then this:

df -hT

and df reported my /opt mount still at 100% used.

Any suggestions?

Answer 1

Restart tomcat, if the file is in use and you remove it, the space becomes available when that process finishes.

Answer 2

Is the rm journaled/scheduled? Try a 'sync' command for force the write.

Answer 3

If something still has it open, the file won't actually go away. You probably need to signal catalina somehow to close and re-open its log files.

Answer 4

If there is a second hard link to the file then it won't be deleted until that is removed as well.

Answer 5

The linux filesystem considers "opened" files to be another name for them... rm removes the "name" from the file as seen in the directory tree... but untill the handles are closed, the files still has more "names"... so the file still exists. The file system doesn't reap files until they are completely unnamed.

Depending on the filesystem, this is usually implemented as a sort of "refrence count".

It might seem a little odd, but doing it this way allows for useful things... like symlinks (multiple names for the same file as seen by directories)

One big benifit is that you can completely wipe out the kernel and all the libraries "from underneath" running programs. But since the "name" still exists for the older copies, the file still exists in memory/disk for that particular program.

It doesn't work so well anymore, but you used to be able to update the entire OS, start up a new http-daemon using the new libraries, and finally close the old one when no more clients are being serviced with it (releasing the old handles) . http clients wouldn't even miss a beat.

Answer 6

As FerranB and Paul Tomblin have noted on this thread, the file is in use and the disk space won't be freed until the file is closed.

The problem is that you can't signal the Catalina process to close catalina.out, because the file handle isn't under control of the java process. It was opened by shell I/O redirection in catalina.sh when you started up Tomcat. Only by terminating the Catalina process can that file handle be closed.

There are two solutions to prevent this in the future:

• Don't allow output from Tomcat apps to go into catalina.out. Instead use the swallowOutput property, and configure log channels for output. Logs managed by log4j can be rotated without restarting the Catalina process.

• Modify catalina.sh to pipe output to cronolog instead of simply redirecting to catalina.out. That way cronolog will rotate logs for you.

Answer 7

As others suggested, the file probably is still opened by other processes. To find out by which ones, you can do

lsof /opt/tomcat/logs/catalina.out

which lists you the processes. Probably you will find tomcat in that list.

Answer 8

lsof did report the file still open by java.

i restarted tomcat and all's well.

thanks everyone!!

Answer 9

Restarting Tomcat will release any hold Tomcat has on the file. However, to avoid restarting Tomcat (e.g. if this is a production environment and you don't want to bring the services down unncessarily), you can usually just overwrite the file:

cp /dev/null /opt/tomcat/logs/catalina.out

Or even shorter and more direct:

> /opt/tomcat/logs/catalina.out

I use these methods all the time to clear log files for currently running server processes in the course of troubleshooting or disk clearing. This leaves the inode alone but clears the actual file data, whereas trying to delete the file often either doesn't work or at the very least confuses the running process' log writer.