What CPAN module can summarize Perl error logs?

Question

I'm maintaining some website code that will soon dump all its errors and warnings into a log file. In order to make this a bit more pro-active I plan to parse this log file daily, summarize the warnings and errors (i.e. count the occurrence of each specific one and group by either warning/error) and then email this to the devs on the project.

This would likely admittedly be rather trivial with a hash and some further fiddling, I wondered if there is a suitable module on CPAN that I could use to do this task.

It would either be one that summarizes specifically Perl error/warnings logs or one that summarizes arbitrary text files. Any suggestions?

Edit:

The site I'm maintaining was inherited in a state where it generates 50MB of warnings per day. I'm ONLY looking for a bandaid i can apply to point out the most prolific ones. Log4Perl can come in once I run out of critical stuff to fix, however right now it's not an option.

Answer 1

Not a CPAN module, but the loganalysis site has some very useful tools and info on log parsing and analysis.

Also, log_analysis may be worth looking at as is implemented in Perl.

Answer 2

There isn't going to be a magical module that handles any log format, including the made up ones people use locally, that you can throw at it. Is there something about your log format? Do you have a printf-style description of it? Does it look like a widely-used format for something else?

If you get to choose the format of the error message, make it look like something that a tool you like can understand.

You might also consider using something like Log4perl. Not only can you specify any format that you like, but you can send the output anywhere you like. You can even send the output to a database, fully normalized, so that your summarizer is really just some SQL.

Update

You clarify in a comment (although you did not edit your question to clarify) that this is for warnings and errors emitted from perl. In that case, it sounds like the developers need a proper test suite to catch all of that stuff. If you're putting stuff into production with the plan of catching warnings then, you have a broken process.

Answer 3

I think looking on CPAN for something as specific and simple as this might be overkill. Assuming the log file in in the default apache error format:

[Mon Apr 26 15:39:34 2010] [error] [client 69.12.220.202] syntax error at /var/www/cgi-bin/errortest.cgi line 8, near "{}"

Here's a quick one liner to mail the errors sorted by highest occurrence to an address. Easily changed to mail multiple addresses (or make an alias that goes to multiple addresses and send to that.

cat LOGFILE |
perl -ple 's/\[\w+\s\w+\s\d+\s\S+\s\d+\]\s\[[^\]]+\]\s\[[^\]]+\]\s//;' |
sort |     # Sort errors after
uniq -c |  # Uniqify with count
sort -rn | # Sort line counts
mail -s "Error list" EMAIL@ADDRESS

You can easily exclude lines by throwing in a grep -v at whatever point you find most appropriate. Throw it into cron for daily reports, or put it into a script and add to logrotate.