A method for high-load web site logging to file?

Question

I need to build in click and conversion tracking (more specific and focused than IIS log files) to an existing web site. I am expecting pretty high load. I have investigated using log4net, specifically the FileAppender Class, but the docs explicitly state: "This type is not safe for multithreaded operations."

Can someone suggest a robust approach for a solution for this type of heavy logging? I really like the flexibility log4net would give me. Can I get around the lack of safe multi-threading using lock? Would this introduce performance/contention concerns?

Answer 1

While FileAppender itself may not be safe for logging, I'd certainly expect the normal access routes to it via log4net to be thread-safe.

From the FAQ:

• log4net is thread-safe.

In other words, either the main log4net framework does enough locking, or it has a dedicated logging thread servicing a producer/consumer queue of log messages.

Any logging framework which wasn't thread-safe wouldn't survive for long.

Answer 2

I'm also interested in the answer, but I'll tell you what I was told when I tried to find a solution.

An easy way around it would be to use something like an SQL database. If the data you want isn't well suited for that, you could have each page access write it's own log file and then periodically merge the log files.

However, I'm sure there's a better solution.

Answer 3

You could check out the Logging Application Block available in the Microsoft Enterprise Library. It offers a whole host of different types of loggers, as well as a handy GUI configurator that you can point to your app.config\web.config in order to modify it. So there's not need to sift through the XML yourself.

Here's a link to a nice tutorial on how to get started with it:

http://elegantcode.com/2009/01/20/enterprise-library-logging-101/

Answer 4

Hi,

When using syslog, you won't be having any threading issues. Syslog, sends the loglines using UDP to a logdaemon (could potentially be on the same machine).

Works especially great if you have more running processes/services, since all log lines are aggregated in 1 viewing tool.

if you expect really heavy loads, look at how the guys from facebook do it: http://developers.facebook.com/scribe/ You can use their opensource logtool. I don't think you'll hit their kind of load just yet, so you should be safe for some time to come!

R