App Server Logs

Question

What are some of the techniques / tools you use to analyze your application server logs?

My dev environment is Windows and my logs are on prod unix boxes .Some times I need to go thru archived logs(atleast on 4 servers) from many months ago to figure out a root cause of a error or exception. It is kind of a time consuming process and I want to hear from the community some of the best practices.

Thanks

Answer 1

Apart from custom scripts there are a variety of tools to help you with this. Lots of very good paid for solutions are available.

One good open source option is chainsaw it's from the log4j developers and is apache licensed:

http://logging.apache.org/chainsaw/index.html

Answer 2

Take a step back and see if you have a log searching problem or an error reporting problem.

Does a single error result in multiple log entries or a single one? Do you have thousands of lines of info and debug messages for each error? Why are your logs so hard to search?

Without seeing your code; is it littered with the following?

} catch (Exception e) {
  //error suppressed 
  log.error("error" + e.getMessage());
}

...

} catch (Exception e) {
  //error logged and passed along
  log.error("error" + e.getMessage());
  throw e;
}

...

} catch (Exception e) {
  //error logged and new one passed along
  log.error("error" + e.getMessage());
  throw new Exception("error" + e.getMessage());
}

The end result is that a single error can lead to multiple error log entries as the problem is logged and bounced rather than handled. I call this bureaucratic logging since all errors are filed in triplicate, passed around, and no one takes actual responsibility in handling the problem.

I would consider separating errors from info and debug messages and work to make reported bugs easier to find.

Answer 3

If you have a large number of logs you could look at a log indexing/search solution. this would enable you to index you log files in real time and allow you to search via keywords for the data that you want. there's a product called Splunk that will be able to help you here:

http://www.splunk.com/

For open source versions see the following previous stackoverflow links:

http://stackoverflow.com/questions/183977/what-commercial-and-open-source-competitors-are-there-to-splunk