How can I rotate and compress Log4perl log files?

Question

From what I can tell neither Log4Perl or any of its related modules in CPAN supports rotate & compression of log files.

Rotation can be accomplished by using:

1. Log::Log4perl::Appender::File
2. Log::Dispatch::FileRotate.

But neither modules supports rotation and compression. (Log::Dispatch::FileRotate has it in its todo list, but it's not currently implemented).

It is possible to do this using the standard Logrotate facility in Linux, by using either Log::Log4perl::Appender::File's recreate_check_interval or recreate_check_signal.

From initial tests, it looks like using Logrotate with the delaycompress option will do the trick - even on a machine with high load, as once the file is moved, log4perl will continue logging to the same filehandle, until the signal is cought.

However, if delaycompress is not used, and there is (even a slight delay) between the compressing of the log file, and the catching the signal by the Perl program, some logging data might be lost.

What do you think? Are there other options we did not think of?

Answer 1

Have you thought about working with the Log::Dispatch::FileRotate's maintainers to add the features its missing and you need? It is open source after all. :)

If you don't want to deal with that yourself, there are various CPAN support consultancies that do that for you.

Answer 2

I contacted the author of Log::Dispatch::FileRotate, as suggested here, and he explained the reason why compression is not yet implemented in Log::Dispatch::FileRotate.

Basically, compressing right after rotation, might block the running process, during the compression which is pretty expensive.

The options suggested were to allow the user of Log::Dispatch::FileRotate to execute an arbitrary application on the file, just after rotation, thus doing it in another non blocking process.

Another suggestion was to have a filesystem trigger (like inotify) trigger the compression when the file is closed to writing by the main process.

Yet another suggestion , is to write the log file compressed through a gzip pipe, or one of the perl gzip modules. This works, but causes some problems (grep/less) won't work. zgrep and zless will work, but zgrep gives an ugly warning when grepping on a gzip file which is still open for writing. Using "tail" on the file will also not work - so this option isn't practical.

Answer 3

Over the years, I've found that you almost always want to use external methods for log file rotation with Log4perl. You simply avoid a lot of subtle issues (log delays, permission issues) that internal log rotates inevitably run into.

You've mentioned two methods that work with logrotate on Linux, why not stick with them? The Log4perl FAQ describes using newsyslog which is the FreeBSD equivalent of logrotate and provides similar features.

Now, if this goofy website wouldn't deny using hyperlinks to new (but registered) users, I would have embedded that link, but since it is, here's the URL: search.cpan.org/dist/Log-Log4perl/lib/Log/Log4perl/FAQ.pm#How_can_I_rotate_a_logfile_with_newsyslog?