Searching huge log files...

Question

Troubleshooting, analyzing & filtering log files is by far one of the most laborious daily jobs. My issue is searching through a log file, which could be well over 4 gigs in size. Simply loading the file takes up to 15 mins. I'm running a fairly fast processor with 8 gigs of memory. After the file loads, I literally only have the luxury of grep and/or control+F to scan through the file. This gets worse when I'm trying to look files from multiple systems each weighing over a gig. Have tried segregating the files based on time-stamps to make them smaller, but no joy really.

Is there a tool or even a process that I could use to make troubleshooting less time consuming (apart from the usual "just fix the bug first")?

Your comments are appreciated.

Answer 1

What are you loading it with? 4 gigs is a fairly large file, but that shouldn't take THAT long to load into memory.

For files that large, I would recommend using grep directly, and if grep isn't doing it for you, SED and AWK are your friends. If you want to do it in realtime, learn about using those tools in conjunction with pipes and tail -f.

Yes, I know, SED is very intimidating at first. It's also ridiculously powerful. Learn it.

If you're on windows, you have my sympathy. May I recommend a unix shell?

If you are afraid of the command line tools, consider learning Perl or Python. They're both quite good at sorting signal from noise in large files like this.

Answer 2

Baretail is a good tool to have. Give it a try. I haven't used it for 4 gigs files but my log files are also quite big and it works just fine. http://www.baremetalsoft.com/baretail/index.php

edit: I did not see that someone has already suggested baretail.

Answer 3

If you want to exclude lines of things you don't want to see, you can grep -v 'I dont wanna see this' > logWithExcludedLines.log. You can use regex as well grep -vE 'asdf|fdsa' > logWithNoASDForFDSA.log

This method works very well with apache access logs grep -v 'HTTP/1.1 200' > no200s.log (or something like that, don't remember the exact string).