Log management&analysis Open Source product

Question

My company is searching for an Open Source product that could do log management and analysis (edit: we'd use the product to handle custom log files - the ones generated by our applications, not standard logs)

We've had demos for some proprietary products (for example, Sensage or HP's Transaction Vision), but we'd like to find an Open Source alternative.

Requirements are:

• The product should be able to collect logs from various machines (Solaris, Windows, etc.)
• It should handle substantially big logfiles (even 500Mb per logfile)
• It should handle different logfiles per day (for example, 24 logfiles a day with different names)
• It should be able to store only the relevant informations from each logfile (for example, some fields, in a configurable way)
• It should be able to handle multi-line records in logfiles (for example, xml)
• Preferred method of storage for log records is an Oracle or MySQL database (not exotic indexing systems) (edit: after seeing Splunk in action, other forms of data storage are acceptable too)
• It should handle quasi-realtime parsing and analysis of log files (10 or 20 minutes of delay is acceptable)

I know it seems too much and too specific for an Open Source system, but maybe Open Source could surprise us again.

Answer 1

I don't think there is anything from open source. What you're asking for is an integrated solution, open source projects aren't usually interested in this, they're mostly creating general purpose frameworks and then it's up to you to put up a solution on top of this. Your description fits perfectly to what we already have built as a commercial not expensive product - logFaces. There are other solutions as well, depending on how much you're willing to spend for integrating it with your system. I'm biased or course, but most of them are complex, non-trivial to setup and will require additional costs for getting it go in real life.

Answer 2

Look into www.ossec.net. I don't know if it has all of your requirements; nonetheless, it's pretty powerful.

OSSEC is an Open Source Host-based Intrusion Detection System. It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.

It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows. A list with all supported platforms is available here.

Answer 3

To me, this sounds like a combination of tools. For managing system logs, I would recommend syslog-ng from Balabit, combined with the Simple Event Correlator to do log analysis.

Syslog-ng uses the syslog protocol, but enhances the capabilities of the daemon dramatically, handling multiple hosts with ease, splitting the incoming syslog messages into almost any set of categories imaginable. It is also possible to process the incoming logs in a variety of ways, including passing them along to another server.

Simple Event Correlator (SEC) is an extremely powerful log analysis tool which includes time oriented analysis. For example, it can raise the alarm when a particluar log message is seen right after another specific entry. SEC can raise the alarm when a particular log entry is not seen, or a particular log entry is seen too much. It can do a wide variety of things, including shell commands, alarms, and more.

Answer 4

Will this help? http://code.google.com/p/dataparksearch/

Answer 5

Try looking at Splunk. That's what they did initially before they became the IT Search thingy.

http://www.splunk.com/

Answer 6

If it where me, I'd recommend a scripting language of some sort. If you're comfortable with perl I'm sure you could write your own script to do it. It runs on windows, linux, mac, *BSD. Totally open source and many projects use it to do the same thing you're doing. Really well documented, HUGE repository of reusable code. The only down side is the company would have to write it or have it written which would take time and/or additional money to do but it would do exactly what you want it to do. Just my 2 cents.

EDIT: So I'm looking around on sourceforge for a solution to your problem and I found this. http://sourceforge.net/projects/lofimo/

what piqued my interest was the fact that it could execute shell commands on certain events. Is this more what you where looking for?

Answer 7

Apache Chainsaw V2 can do much of what you need:
    * The product should be able to collect logs from various machines (Solaris, Windows, etc.)
      Yes, by configuring a VFSLogFilePatternReceiver, which uses Jakarta Commons-VFS supported file systems
    * It should handle substantially big logfiles (even 500Mb per logfile)
      Yes, limited by how much memory you allocate to the VM, but Chainsaw also supports cyclic buffering if needed as well as the ability to filter events to reduce memory footprint (mentioned below)
    * It should handle different logfiles per day (for example, 24 logfiles a day with different names)
      You can define any number of receivers (one receiver/one log file) in a Chainsaw configuration, but it won't watch for log files in a folder and pull them in dynamically (but you could write a custom receiver to do that)
    * It should be able to store only the relevant informations from each logfile (for example, some fields, in a configurable way)
      You can define a filter expression on the VFSLogFilePatternReceiver, which will result in only matching events being processed
    * It should be able to handle multi-line records in logfiles (for example, xml)
      Yes, VFSLogFilePatternReceiver supports multiple line records, but it doesn't support XML - there is a LogFileXMLReceiver, but that receiver doesn't support Jakarta commons VFS file systems, and you'd have to write your own 'xml decoder' which understands how to parse your xml
    * Preferred method of storage for log records is an Oracle or MySQL database (not exotic indexing systems) (edit: after seeing Splunk in action, other forms of data storage are acceptable too)
      You can have any received events processed by an appender (DBAppender, for example, to have received events saved to a db automatically), or you can export events  as xml
    * It should handle quasi-realtime parsing and analysis of log files (10 or 20 minutes of delay is acceptable)
      Yes, it will parse and tail log files in realtime

Chainsaw also supports a number of other useful features: filter, colorize and search based on event contents using an expression syntax, annotate events and save the annotations with the events, save & load the information later with the annotations (exported as xml)

Feel free to email or ask questions on the mailing list.

Answer 8

Hi check out XpoLog Center and XpoLog TxExpo www.xpolog.com The product can do the analysis and transaction analysis on logs.