What is the best log analysis tool that you used?

Question

I'm starting to work on an existing project that has very verbose logging and the logs are a real pain to go through. Have you guys ever used any tools for log analysis/parsing? Can you recommend me something powerful and easy to customize and use? At first I thought I should write a tool to parse the logs, but I'm thinking maybe something like that already exists. Thank you for the help in advance.

Edit: The logs are custom - they have a set format, but I don't think they conform to any widely used standards. That's why I need a customizable log parser.

Answer 1

Log Parser from Microsoft

http://www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspx

Free, and works incredibly well.

Answer 2

We use Splunk here. http://www.splunk.com/

It's very powerful, allowing multiple log sources, templating, filtering, etc. It's a little more than a "single-user" solution, but they boast a 5-minute install.

Answer 3

For IIS Logs, I tend to use Microsoft Excel. Import the file as space delimited, delete the header rows and the #Fields: label and then use the AutoFilter functions to slice and dice the data as you need it...

Answer 4

I agree - LogParser.

Answer 5

Analog ...its free, fast, and very easy to use.

Answer 6

If you are going to use LogParser you might find this article valuable.

Answer 7

Log Parser is an excellent tool for this.

You might want to check out Visual Log parser over on Codeplex, though it doesn't look like much work has been done on if for awhile.

There's a forum here that discusses Log Parser.

Also, Mike Gunderloy had an unofficial FAQ here, and here's a nice article over at Coding Horror

Answer 8

You can try Octopussy (it's my project)

It is a log analysis/alerting/reporting solution made to handle any kind of logs. It's quite complicated at the beginning, but after, it's really powerful. It's fully customizable and you have a 'logs wizard' to create new parsers.

Answer 9

Currently experimenting with SED and MySQL. SED is used to solve syntax level issues and convert to CSV. The SED script took a few hours (including learning SED), but I expect the hard part to be the analysis in MySQL of course.

I just found that none of the downloadable tools were able to permanently categorise hits and consistently include/exclude the correct categories in reports.

If you are thinking of improving this custom log format of yours then log4x with the Xml layout is one to consider. Take note of the NDC and MDC output. Of course, you'd need to switch to log4x too. Your analyser in this case would be Chainsaw, but you are dealing with XML fragments so a custom solution is also possible.

Answer 10

SLCT - Simple Logfile Clustering Tool figures out recurring patterns on its own and prints a concise summary. It's good for a quick look at a log for which you don't have any parser or grammar yet.

Answer 11

Perl

It usually takes a very short time to write a custom log parser.

Answer 12

Apache's Chainsaw V2 can parse (and tail) text log files, using LogFilePatternReceiver or VFSLogFilePatternReceiver, if you need Jakarta Commons-VFS supported file systems (for example, tailing over ssh). See http://logging.apache.org/chainsaw - there is an example configuration available from the Welcome tab and a simple tutorial available from the help menu.

Answer 13

XpoLog Log Analysis platform provide reports, search, monitor, statistics, trends, error detection and much more for web server and applications logs www.loganalysis.com

Answer 14

XpoLog works great for us XpoLog