multithreaded logging for high performance application

Question

I have an application (server application) that needs an extensive amount of logging implemented and shouldn't be too affected performance wise by enabling logging.

The application has a thread pool of worker threads performing the work. Originally I was going to just log on these thread pool threads, but then I'd need to lock practically the whole thread and so there goes my 'multithreaded' app.

I have been looking at better ways to log from multiple threads and I've found using a Queue or ring buffer could be an idea.

Can anybody suggest (perhaps from experience) any good ways to implement effective logging (to a file mostly) for a multithreaded application that also should stay somewhat performant?

I would like to use the Boost Logging Library.

Answer 1

Pantheios is the fastest logging library for C++ out there, as far as I know. I recommend using it instead of Boost Logging. With Pantheios you simply log to the file, and you don't care from which thread. You can put the thread name in the logline prefix if you want and it does everything for you.

Answer 2

Personally I would look into Pantheios, gave it a glance and it seems interesting, going to include that in a future project of mine.

If you really want to use boost logging I would use a synchronized queue that handles all the locking internally so your workers doesn't have to worry about that.

pseudocode:

 class SynchedQueue {
      void write(text) { lock() logfile.write(text) unlock() }

Or if you really want to make it fast, use an internal queue and transfer from the public queue in batches of X lines at a time. The public queue doesn't have to wait for file i/o that might take a relativily long time and the private queue only gets lines seldomly. Although it will probably still be slower than Pantheios.

Answer 3

I read an interesting article using Active Objects for mt logging on ddj recently:

http://www.drdobbs.com/high-performance-computing/227500074

An older article also discussed some design considerations for thread-safe logging:

http://www.drdobbs.com/cpp/201804215

Answer 4

Apache log4cxx. I have seen quite a lot of places where it is being used extensively. And most of them were trading applications, multi threaded and low latency applications. log4cxx isn't inferior to any logging library and it is available for C++ and Java(Apache log4j), these 2 I have used.

Answer 5

I'm not familiar with the Boost Logging library, so I can't comment on that.

NLog supports logging from C/C++ and COM (as well as from managed .NET applications). Not sure if it has the performance characteristics you are after or not.

You might also consider ETW. It is a high speed logging system implemented in the Windows kernel. I have not used it but Microsoft does seem to be pushing it a little more and making it a little easier to use with each release. They are certainly making it easier to use from managed code. As it stands right now, it looks to me to be somewhat complicated to get set up and then to get the output. I have never gotten to the point that I have seriously considered implementing ETW, so I can't really comment on how easy or difficult it really is.

Answer 6

I am using and really liking Petru Marginean's logging system. It is really light fast and neat. It took me a little while to get my head round it so I could adapt it to my liking, but once you understand it, it's a beauty and as the whole library is only really 1 header file of 250 lines, its not much to get through.

The slowest part of it is it uses an object with an ostringstream, but I doubt you will find anything faster. At least not without being more C than C++.

In his second article and version he shows how he made it thread safe.

If you cant find the source code, let me know, I have it. I remember I had to search for it a bit.