Performance comparison for logging to MSMQ / text file / database

Question

We have several choices for logging in our .NET (C#) server application. We are going to use Enterprise Library. So here are the ways to go:

1) Writing log to MSMQ synchronously, then reading MSMQ with Win Service. Queue is on the local machine for server application.
2) Writing log to disk (i.e. rolling text files) synchronously.
3) Writing log to database (Oracle, in our application) synchronously.

Log amount might be fairly high. So which one is the most performant? I guess ordering is 1, 2, 3. Am I right? Is there any other performance factor, besides write speed, in this specific scenario? Are there any other choices, that I have not pointed out here, that might be more better way?

Answer 1

Without hearing the application's needs in terms of logging:

I get the feeling that the MSMQ logging scenario is a bit of a large hammer to the problem.

MSMQ definitely will be faster, as your application will be using a network communication protocol instead of dealing with disk latency and DB connection creation & tear down. Given though, that MSMQ uses the disk, the gain is likely negligible. Then the question is whether that MSMQ target/destination queue is on the local machine.

Stick with the Disk Logging

Consider sticking with logging to a file on disk with appropriate rollovers (day/hour as required). There are some workarounds if you're SUPER concerned with spindle speeds - consider writing those log files to a RAM disk. Have then a process to copy those files out of RAM and onto disk periodically as you see fit.

Measure Your Logging Performance Needs

Cue the jokes about premature optimization. Measure, and measure again. You just won't know what you need until you gauge how well your solution is performing. Consider that you likely have 10,000rpm spindles, and likely a disk logging solution will perform adequately. A bit of YAGNI might creep in if the decision is to jump to the complex solution of an 'outsourced' component like MSMQ or a database write. I say this with your stated needs around speed.

Answer 2

I think you should consider "Write log to database asynchronously". i.e. for each item you want to log, queue it up in memory and have a separate thread writing to the database. No sense keeping the main application waiting for logging to complete provided the order of the logging messages is maintained by the queuing mechanism.

Also, did you look at log4net? It includes numerous logging options including a powerful rotating log to disk, UDP logging, ....

You might also want to look at Splunk as a way to search your log files after making them.

Answer 3

You're ranking of MSMQ, File, and DB should be correct in typical situations.

I know you're concerned about performance but I have yet to encounter a scenario in a line of business application where I've regretted the logging mechanism used. When compared with the amount of time spent performing the business logic, the time spent logging was not just significant.

Also, if you are really concerned about performance you may want to investigate other options besides Enterprise Library. e.g. log4net has historically run faster.

In addition to performance, I would also consider other requirements. e.g. Do you need to report or query on the logs? Do you require a central log repository (or would it be useful)? If the answers to any of those questions is yes, then you would probably want to go with a database or MSMQ feeding a database.

Other considerations might be existing implementations or standards, overall solution complexity (MSMQ > file), supportability of the various options (e.g. you may not want to use MSMQ if there is no operational expertise).