"Noisy" Logging Problem.

Question

Throughout our enterprise we have many applications that log activity to text files for diagnostic purposes. Generally if something fails or breaks an email notification is sent to notify our team of problems. Also we have a monitoring application that notifies our team via email when an application has a longer than expected period of inactivity. The problem we are facing is that the logs have devolved into noise, meaning that they fail to provide quick information about failures and successes. I am currently exploring a solution specific to the general architecture of the applications in our enterprise which leads me to my question:

What is are some common approaches, techniques and possibly even frameworks that can provide concise logging information that is quickly viewable and readable?

Answer 1

Log4J and other logging frameworks provide the concept of log levels, where you specify a priority with each log statement. Then at runtime, you set the level of the log file itself - only log statements with that priority or higher will get through. So for example, you might set up your production system to only log messages at levels WARN or ERROR, while your development environment can be set to log anything at DEBUG or higher.

This is a nice solution because it doesn't require any change to the code (like adding/removing log statements for each new build), and can easily be reconfigured (temporarily) when you need to do something like debugging a specific problem.

If your system is using the .NET architecture, there is a port of log4j called log4net that you could use (though obviously, replacing your existing system would require a code change). However, this question asked about general design principles, not about what to do with an existing codebase, so I think this answer suffices.

Edit: since you have the luxury of using a brand new framework, this page has a list of alternatives, though I am partial to log4j due to my own use in Java applications.

Answer 2

small and quick answer, since it seem to be all internal product, create a database and log everything in it, make the software that would query that database so it would give more information that just an email.

Answer 3

I recomment log4net. You may need to build own warpper around of it (more frenly to use).

Answer 4

Got Splunk?

Answer 5

If you're looking to correlate multiple logs, Splunk is definitely worth a look.

For making your application log data more accessible and easy to interpret, take a look at Gibraltar.

Gibraltar includes powerful log analysis tools that can help you see patterns across thousands of logs. While it includes its own logging API, it also works with popular logging frameworks such as log4net, NLog, the ObjectGuy's DotNetLog as well as the Trace/Console classes in the .NET System.Diagnostics namespace.