Logging strategy vs. performance

Question

Hi,

I'm developing a web application that has to support lots of simultaneous requests, and I'd like to keep it fast enough. I have now to implement a logging strategy, I'm gonna use log4net, but ... what and how should I log? I mean:

1. How logging impacts in performance? is it possible/recomendable logging using async calls?
2. Is better use a text file or a database? Is it possible to do it conditional? for example, default log to the database, and if it fails, the switch to a text file.
3. What about multithreading? should I care about synchronization when I use log4net? or it's thread safe out of the box?

In the requirements appear that the application should cache a couple of things per request, and I'm afraid of the performance impact of that.

Cheers.

Answer 1

1. It makes thing slower - doing something takes more time than doing nothing. Usually by a negligible amount. Don't worry about it.
2. Log to a text file imo. They're easy to move/grep/compress/mail etc. and you don't have to worry about logging to a database the fact that the database is down. There's appenders for logging to a database for log4net if you need that though.
3. Yes, log4net is thread safe.

Having logging/tracing is extremely valuable - at the very very least you should log errors, or you'll never know about them. Most logging apis let you turn on and off the level of detail you need logged.

Don't worry about performance until it becomes a problem. It's not like you're building moon rockets and wants to see how much weight it can carry by testing it - it's just code, remove the logging statements that flood your logs and recompile if it ever becomes a problem.

Answer 2

I would say you are worrying about performance to early on in the piece, use log4net in a way that is very easy to turn off later or customize (eg: use of conf file that defines log level NONE, ERROR, WARN, DEBUG, INFO, ALL, VERBOSE etc ...),

However your two other concerns are valid, for question 2, I would go for plain file as they can be easily read anc accessed as opposed to database. Also the performance of writing to the end of a file is better than a db.

And yes as nos has stated log4net is threadsafe.

One piece of general advice, If your project is a large and you want to have any hope of ever having configurable log levels as I described above, then you really need to agree on a coding standard within your team on what to log at different levels.

Answer 3

I agree completely with what hhafez and nos said. You're going down the right path with a logging package instead of trying to roll your own. It's much cleaner and easier to get right. Logging to a text file is much easier to manage long term (given typical project skill sets) than DB logging, though if you're planning any complex analysis of reported data, sometimes it's easier just having it already in a DB.

If debugging is one of your stated objectives for implementing a logging solution, then it's imperative that you standardize all your log levels up front and make that part of your code review process. Have enough differences in granularities so that you can gradually increase the depth of reporting by going to the next level. It's very frustrating to be troubleshooting a PROD problem, not have enough log info to see the problem, then increase to the next level of logging and completely swamp the logs with so much spew that you can't see the forest for the trees (and your logs roll every 5 minutes because of the volume). I've seen it happen.

In most cases of text file logging, performance should not be an issue. It's a little trickier with DB logging. Doing an insert is only slightly more intensive than appending to a text file, but it's the volume-per-unit-time that makes it much uglier at scale.

Also, if you're going to do any offline log analysis, you ought to pick a log file format that's easily extensible and won't require tremendous changes to the analysis code if you need to add something to the log. Stay away from nested, multi-part message structures. Parsing those gets to be a pain.

Good luck with it!

Answer 4

Thanks all! very instructive answers :D