Minimal overhead distributed event logging library/framework?

Question

We'd like to keep records of all major events in our systems. For example, where the database might store the current user status, the event log should record all changes to that status along with when the changes occurred.

The event logging facility should be as close to zero overhead for the event thrower as possible, should accommodate structured information (as opposed to text log messages), and should support distributed deployment (many boxes throwing many events).

In a past life we had a UDP based system that worked well because we had great control over the system (minimized packet loss). The even throwers would fire off UDP packets that would be caught and journaled on other boxes. I'm looking for something similar, hopefully open source, off the shelf, and deployable in more general networks. Alternatively I'm open to suggestions for how to build something like this.

This should work across multiple languages, but will be primarily targeted for Java and Python. The pariticipating (event throwing) applications will vary; some will be web apps, others batch oriented apps. The results will likely live in Hadoop/HDFS/HBase.

Answer 1

If you want to go down the UDP route (as you seem happy with that), and Java is an option, then check out Log4j and its support for UDP transmission via the Log4j UDPAppender.

LoggingEvent will take a java.lang.Object as a message, so it's pretty generic and you can throw whatever data you want into that. If you're going across the network it should (most likely) be serialisable, and given that you want UDP, should be of a comensurate size - 64k or less, and then dependent on the transport layer). You'll simply have to intercept the LoggingEvent on the server side and then process it however you want.

Note that the UDP appender comes as a companion component to Log4j and you'll have to build it yourself. But that's trivial.

Answer 2

It sounds like a potential candidate for messaging (fire and forget). I'm a .NET person mostly so don't know what logging frameworks there are out there for Java. But I had a quick look to see if there are any messaging appenders for log4j (I use log4net quite often)- IBM have an article on a WebSphere MQ JMS appender, which might be helpful to you.

So rather than take my answer as advocating the use of WebSphere MQ- please take as a suggestion to consider messaging- there are lots of open source messaging frameworks out there- RabbitMQ is just one example.

Answer 3

You may consider using old good *nix Syslog. It has very small overhead and is mostly used over UDP or local UNIX sockets, but may use TCP if you need reliable logging. Works for my (Python/Perl, mostly, but it is completely language/platform-agnostic) like a charm.

Sorry, I'm not familiar with Java, but feature-wise, this seems to be some good library I've googed: http://syslog4j.org/

Edit: Quick googling discovered an article called "Robust event logging with Syslog", which seems to be pretty detailed on the subject. Sorry, I've misread it when posted and thought it is a *nix syslog library, but it isn't.