Rotating logs without restart, multiple process problem

Question

Here is the deal: I have a multiple process system (pre-fork model, similar to apache). all processes are writing to the same log file (in fact a binary log file recording requests and responses, but no matter).

I protect against concurrent access to the log via a shared memory lock, and when the file reach a certain size the process that notices it first roll the logs by:

1. closing the file.
2. renaming log.bin -> log.bin.1, log.bin.1 -> log.bin.2 and so on.
3. deleting logs that are beyond the max allowed number of logs. (say, log.bin.10)
4. opening a new log.bin file

The problem is that other processes are unaware, and are in fact continue to write to the old log file (which was renamed to log.bin.1).

I can think of several solutions:

1. some sort of rpc to notify other processes to reopen the log (maybe even a singal). I don't particularly like it.
2. have processes check the file length via the opened file stream, and somehow detect that the file was renamed under them and reopen log.bin file.

None of those is very elegant in my opinion.

thoughts? recommendations?

Answer 1

1. What about opening the file by name each time before writing a log entry?

   1. get shared memory lock
   2. open file by name
   3. write log entry
   4. close file
   5. release lock

2. Or you could create a logging process, which receives log messages from the other processes and handles all the rotating transparently from them.

Answer 2

You don't say what language you're using but your processes should all log to a log process and the log process abstracts the file writing.

Logging client1 ->  |
Logging client2 ->  |
Logging client3 ->  | Logging queue (with process lock) -> logging writer -> file roller
Logging client4 ->  |

Answer 3

You could copy log.bin to log.bin.1 and then truncate the log.bin file. So the problems can still write to the old file pointer, which is empty now.

See also man logrotate:

   copytruncate
          Truncate  the original log file to zero size in place after cre‐
          ating a copy, instead of moving the old log file and  optionally
          creating  a new one.  It can be used when some program cannot be
          told to close  its  logfile  and  thus  might  continue  writing
          (appending)  to  the previous log file forever.  Note that there
          is a very small time slice between copying the file and truncat‐
          ing it, so some logging data might be lost.  When this option is
          used, the create option will have no effect, as the old log file
          stays in place.

Answer 4

Since you're using shared memory, and if you know how many processes are using the log file. You can create an array of flags in shared memory, telling each of the processes that the file has been rotated. Each process then resets the flag so that it doesn't re-open the file continuously.

Answer 5

Your solution seems fine, but you should store an integer with inode of current logging file in shared memory (see stat(2) with stat.st_ino member).

This way, all process kept a local variable with the opened inode file.

The shared var must be updated when rotating by only one process, and all other process are aware by checking a difference between the local inode and the shared inode. It should induce a reopening.