Logging from multiple apps/processes to a single log file

Question

Our app servers (weblogic) all use log4j to log to the same file on a network share. On top of this we have all web apps in a managed server logging errors to a common error.log. I can't imagine this is a good idea but wanted to hear from some pros. I know that each web app has its own classloader, so any thread synchronization only occurs within the app. So what happens when multiple processes start converging on a single log file? Can we expect interspersed log statements? Performance problems? What about multiple web apps logging to a common log file? The environment is Solaris.

Answer 1

This is generaly bad idea to have not synchronized write access to a file and certainly bad programming practice. The only case it might work is an append to a file on local machine - everybody just adds lines at the end of file.

But, since your file is on the network share, it will probably quickly turn into garbage. You didn't tell which distributed filesystem you are using, but for NFS you can find following explanation on open(2) man page:

O_APPEND The file is opened in append mode. Before each write(), the file offset is positioned at the end of the file, as if with lseek(). O_APPEND may lead to corrupted files on NFS file systems if more than one process appends data to a file at once. This is because NFS does not support appending to a file, so the client kernel has to simulate it, which can't be done without a race condition.

Of course this is C, but since Java is implemented in C it cannot do any better than that (at least not with regard to system calls:-)).

Answer 2

We use org.apache.log4j.net.SyslogAppender to log to a single machine using syslog, and it has worked well for us. I would recommend looking into that as an alternative.

Answer 3

This looks like a really bad idea (corrupt logs, uncertainty of the source of a given log entry are two reasons that come to mind). If you are using Log4j in Weblogic like this, I suggest doing it by-the-book. This will allow you to use one file for the whole app server without any issues.

The suggestion of syncronizing the log writing makes no sense to me, as you would be basically blocking all applications in the app server when they write a log. If logging is frequent, that will slow everything down significantly.

As for multiple app servers, you need to use something other than file based logging if you want them all consolidated. There are a few ways to do this, one is to log to different files and have a different process combine them, but the better option is probably to use a network based logging repository, using Log4j's SocketAppender or some other method (nathan mentions SyslogAppender which is great if you want a Syslog) to ensure that the file access does not get corrupted.

Answer 4

Best case I would imagine you have a potential performance problem with sync access to the log file resource. Worst case would be some of the scenarios you mention.

I would ask two Qs: a) what was the desired goal with this config and b) can you find a better technical solution to achieve the goal.

My guess is that you have a sysadmin who wants to get a "single log file for the system." The file is thrown on the network share to make it easy to get to. The better answer for the goal might be multiple files, local to each system and something like http://www.splunk.com/ for a nice monitor.

Answer 5

If at all possible use a different file for each instance. This will give the best result with the least effort.

The logback alternative to log4j has a prudent mode to its log writer which explicitly jump through hoops to ensure that new things are written at the end of file. I don't think that will work on network shares then.

If you MUST have a central logging location, then consider setting up a server accepting log events and writing them to the appropriate files. This will ensure it is only a single process actually accessing the file system, and will let the JVM help all it can in terms of synchronization etc.

Answer 6

In prudent mode logback will safely handle multiple JVMs possibly on different hosts writing to the same network-shared file. It can even deal with temporary network failures. Performance should be quite acceptable for few nodes, say 4 or less. For 5 or more nodes all logging heavily you may notice a performance hit.