How to reduce redundant log messages for a web-app?

Question

Does anybody have any advice on how to minimize my logs for a web application?

Right now, I'm logging every error. So if there is a situation where an error occurs on every request (a db connection problem for example), it might get logged for every user on every request. As you can imagine, it doesn't take too many users doing too many things to start amassing a huge log file with redundant log errors.

It has occurred to me to create some type of flagging mechanism to not log the same error within N minutes. I don't like this for 2 reason; 1) it could hide the most significant problem amongst the others and 2) cascading errors may not log their error trail properly, omitting previously logged errors, potentially throwing off my initial diagnosis.

I'm sure I'm not the first person to have this issue ... does anybody have any advice on how to reduce redundant log messages for a web-app?

Answer 1

You could create a logging system that logs to a DB, and have a flag set to determine whether to log individual entries, or just tally a counter for a base log entry. By tallying, you see the gross number of errors, but don't have an exploding log file.

Answer 2

You could try comparing the last entry with the error you're about to put in the log. If this is the same type and source, append a string "(repeated n times)" at the end.

Every time the logger sees such a string and the errors match, it can increase the n until another type of error comes in.

Answer 3

I've combined it with my user error message, so when the error message is shown the user, they can either click, OK or Send to Support. Then the complete error, call stack, & complete request information is entered into the bug tracker.

This hasn't been released yet, so we've yet to see if this is a good idea ... will keep you posted.

PS-I've answered my own question as a for the record, and selected it since the other answers did not meet my needs.

Answer 4

To answer the question you had in the comment, to my knowledge there are no out of the box solution that encompasses this type of functionality.

One other idea to consider (I realize at this point you've accepted your own answer) would be to assign an ID to each error logging point in the system. When the error is logged, you can keep a simple hash table of the error Id and what time the error was logged. Then when processing the logging, if the error already exists in the hash and it has been within 5 minutes (or whatever time you decide) then you can simply abort the logging and update the timestamp.

I'm not sure if having the users send the error message request will resolve your issue, because if all your users submit the message, it'll be just the same as the error logs, only your inbox will get flooded as opposed to simply monitoring the log file itself.