What's the best way to view/analyse/filter huge traces/logfiles?

Question

this seems to be a reoccurring issue: we receive a bug report for our software and with it tons of traces or logfiles.
since finding errors is much easier when having a visualization of the log messages/events over time it is convenient to use a tool that can display the progression of events in a graph etc. (e.g. wireshark (http://www.wireshark.org) for analyzing network traffic)

what tool do you use for such a purpose?

the problem with most tools i used so far is that they mercilessly break down when you feed them huge data traces (> 1GB) so some criteria for such a tool would be:

• can deal with huge input files (> 1 GB)
• is really fast (so you don't have to get coffee while a file is loading)
• has some sort of filtering mechanism

Answer 1

Since every log files have their own format, I afraid there is no off-the-shelf components that fulfill your needs; you have to write your own visualizer.

Answer 2

Good thread for this. Tim Bray (formerly at Sun now at Google) had a contest a few years back on writing scripts to munge gigs of log files: http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder

IMHO mpi4py would now be the optimum solution in terms of implementation time, run time, scalability.

Also, think about going oldskool with AWK. In a recent contest it beat out C/C++/Java, http://anyall.org/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/