I am collecting logs from several custom made applications. Each application has it's own log format. What I'm looking for is a central tool which would allow me to search through all of my logs. This means the tool would have to be able to define a different regex (or alike) for each log file (marking where a record begins, ends, and what are the fields). I've been trying Splunk, but I'm not happy with it, since performance are slow, I'm limited (free version) with the amount of indexed data per-day, and it's not as flexible as I want it to be. Could you recommend a software (preferably free or cheap) for the task?
+3
A:
You can try Lucene. It is free. It is written in Java, and it allows full-text search over large amount of data. It is not a complete application, but rather a library, so you have to write code that uses it to index and to search your logs. You may have to define different document types or at least different indexing functions for your logs, but then search works beautifully.
Yuval F
2008-10-07 09:00:50
You can use solr in addition to Lucene
Vinko Vrsalovic
2008-10-07 09:12:42
A:
If you can use Windows, try out Microsoft's best tool ever, Logparser. I wish there was such a simple tool for Unix. But there isn't. And although I've kept wanting to get around to making a Unix version of Logparser, I just haven't had the time.
Note: This would be a great project for someone with time on their hands or for a grad-student somewhere!
Robert Gould
2008-10-07 09:18:15