Can syslog Performance Be Improved?

Question

We have an application on Linux that used the syslog mechanism. After a week spent trying to figure out why this application was running slower than expected, we discovered that if we eliminated syslog, and just wrote directly to a log file, performance improved dramatically.

I understand why syslog is slower than direct file writes. But I was wondering: Are there ways to configure syslog to optimize its performance?

Answer 1

Write your own syslog implementation. :-P

This can be accomplished in two ways.

1. Write your own LD_PRELOAD hook to override the syslog functions, and make them output to stderr instead. I actually wrote a post about this many years ago: http://marc.info/?m=97175526803720 :-P
2. Write your own syslog daemon. It's just a simple matter of grabbing datagrams out of /dev/log! :-P

Okay, okay, so these are both facetious answers. Have you profiled syslogd to see where it's choking up most?

Answer 2

Before embarking in new daemon writing you can check if syslog-ng is faster (or can be configured to be faster) than plain old syslog.

Answer 3

You can configure syslogd (and rsyslog at least) not to sync the log files after a log message by prepending a "-" to the log file path in the configuration file. This speeds up performance at the expense of the danger that log messages could be lost in a crash.

Answer 4

You may configure syslogd's level (or facility) to log asynchronously, by putting a minus before path to logfile (ie.: user.* [tab] -/var/log/user.log).

Cheers.

Answer 5

One trick you can use if you control the source to the logging application is to mask out the log level you want in the app itself, instead of in syslog.conf. I did this years ago with an app that generated a huge, huge, huge amount of debug logs. Rather than remove the calls from the production code, we just masked so that debug level calls never got sent to the daemon. I actually found the code, it's Perl but it's just a front to the setlogmask(3) call.

use Sys::Syslog;
# Start system logging
# setlogmask controls what levels we're going to let get through.  If we mask
# them off here, then the syslog daemon doesn't need to be concerned by them
# 1   = emerg
# 2   = alert
# 4   = crit
# 8   = err
# 16  = warning
# 32  = notice
# 64  = info
# 128 = debug
Sys::Syslog::setlogsock('unix');
openlog($myname,'pid,cons,nowait','mail');
setlogmask(127); # allow everything but debug
#setlogmask(255); # everything
syslog('debug',"syslog opened");

Not sure why I used decimal instead of a bitmask... shrug

Answer 6

There are several options to improve syslog performance:

• Optimizing out calls with a macro

   int LogMask = LOG_UPTO(LOG_WARNING);
   #define syslog(a, ...) if ((a) & LogMask ) syslog((a), __VA_ARGS__)
  
  
   int main(int argc, char **argv)
   {
            LogMask = setlogmask(LOG_UPTO(LOG_WARNING));
            ...
   }
  

  An advantage of using a macro to filter syslog calls is that the entire call is reduced to a conditional jump on a global variable, very helpful if you happen to have DEBUG calls which are translating large datasets through other functions.

• setlogmask()

  setlogmask(LOG_UPTO(LOG_LEVEL))
  

  setlogmask() will optimize the call by not logging to /dev/log, but the program will still call the functions used as arguments.

• filtering with syslog.conf

   *.err                                               /var/log/messages
  

  "check out the man page for syslog.conf for details."

• configure syslog to do asynchronous or buffered logging

  metalog used to buffer log output and flushed it in blocks. stock syslog and syslog-ng do not do this as far as I know.