Tool or language to count occurrances of errors in a log file

Question

I am trying to determine the best way to parse a log file and get a count of all of the errors in it by type. Currently, I open the log in a text editor, strip out the date and thread ID, then sort the file. This puts all errors together by type, which I can then count (using the count function in the editor, not manually). I am looking for a way to do this automatically, and possibly use this as an opportunity to learn a new language (I know minimal Perl and Ruby which seem like they may work for this task). The log file looks like (the items in angle brackets are variable for each line, while the pipes are actual characters in the log):

<Datetime stamp> | <Thread ID> | ERROR | Foo.Bar: Backend error
<Datetime stamp> | <Thread ID> | ERROR | Foo.Bar: InvalidUserException
<Datetime stamp> | <Thread ID> | ERROR | Foo.Com: Timeout error
<Datetime stamp> | <Thread ID> | ALWAYS | Foo.Bar: Login Transaction [584] executed in [400] milliseconds
<Datetime stamp> | <Thread ID> | ALWAYS | Foo.Bar: Login Transaction [585] executed in [500] milliseconds
<Datetime stamp> | <Thread ID> | ALWAYS | Foo.Bar: Login Transaction [586] executed in [500] milliseconds
<Datetime stamp> | <Thread ID> | ALWAYS | Biz.Dee: Logout Transaction [958] executed in [630] milliseconds
<Datetime stamp> | <Thread ID> | ERROR | Foo.Bar: Backend error

I don't want to use a series of grep commands because I will have to know what to look for - if there is a new error in the log, without adding a new command, it won't get counted.

The output I am looking for is something like this:

Foo.Bar: Backend error: 2 occurrences
Foo.Com: Timeout error: 1 occurrence

Ideally, it would be great to also have the average transaction times calculated as well:

Foo.Bar: Login Transaction: 3 occurrences with an average of 466 milliseconds
Biz.Dee: Logout Transaction: 1 occurrence with an average of 630 milliseconds

I've seen some tools mentioned in other SO threads (SMTP log parser, Microsoft log parser, Zabbix, and Splunk), but I would also like to learn something new without unnecessary duplicating an existing tool. Would Perl or Ruby be a good choice for this task? I am not looking for a working script, but a few pointers in the right direction or a good tool to use.

Answer 1

Perl would be my first choice for the string parsing. Using a RegEx you could parse through that log file in no time. From what I can see it looks like you're dealing with a nicely computer readable file. You could use a Perl hash to do your averaging.

You could likely do the same thing with C# and their RegExs if you're more familiar with that, but Perl was built to do stuff like this.

Answer 2

I would use a RegEx and count the number of occurrences. You can do this in a myriad of languages, even a simple shell script would do it, e.g.

grep -E ".*ERROR.*\n" logfile | wc -l

Answer 3

If you know/like .NET, the Push LINQ framework that Marc Gravell and I developed would be an ideal candidate for this. Basically, you set up all the aggregation you want (grouping, summing etc) beforehand, and "push" the logfile through it, then ask for the results at the end. This would let you do everything with near-constant memory consumption and a single pass through the data.

Let me know if you want more details.

Answer 4

Here's a unix (or Cygwin) command-line way do this with:

• An AWK command (to parse out the 4th field, where your fields are separated by pipes "|")
• A SED command to replace the transaction # ([584]) above to make grouping easier (with [tid])
• sort and uniq to find and count duplicate lines:

Here is the command-line:

awk "FS=\"^|\";{print $4}" logfile.txt | sed -e "s/\[[0-9]*\]/[tid]/g" \
| sort | uniq -c | sort

Here's the output:

   1  Biz.Dee: Logout Transaction [id] executed in [id] milliseconds
   1  Foo.Bar: Backend error
   1  Foo.Bar: InvalidUserException
   1  Foo.Com: Timeout error
   3  Foo.Bar: Login Transaction [id] executed in [id] milliseconds

Answer 5

Here's a possible Perl starting point for you:

#! /usr/bin/perl
use strict;
use warnings;

my %unique_messages;
while (<>)
{
  my ($timestamp, $thread, $type, $message) = $_ =~
    /^
      ([^|]+) \|
      ([^|]+) \|
      ([^|]+) \|
      (.+)
     $/x;

  $unique_messages{$message}++ if $type =~ /ERROR/;
}

print $unique_messages{$_}, ' -> ', $_, "\n" for keys %unique_messages;
exit 0;

Produces:

% ec.pl < err.log
1 ->  Foo.Com: Timeout error
1 ->  Foo.Bar: InvalidUserException
2 ->  Foo.Bar: Backend error

Answer 6

Another possibility using awk:

grep ERROR filename.log | awk -F'|' '{ print $4 }' | awk -FS=':' '{count[$1]++}END{for(j in count) print j,": "count[j]" occurence(s)"}'

Answer 7

You can use a program like monarch to give structure to flat data. I have used it to take text files and make tables out of them that I can use in a database.

Answer 8

microsoft log parser if you are ok with SQL. And using Windows. Free and quite handy. Easy to wrap in an HTA, then you can use VBS or (?) JS to build query strings interactively. Believe it will do subtotals for you. Certainly sorts and groups.

Answer 9

In vim you can do :%s/pattern//n where pattern is the search string.