Centralized network logging - syslog and alternatives?

Question

At work, we're building a distributed application (possibly across several machines on a LAN, possibly later across several continents on a WAN+VPN). We don't want log files local to each machine (filling up its disk and impossible to view in aggregate) so we need to centralize logging over the network. Most logs won't be important, so UDP is fine for them, but some are money-losing important alerts and must be reliably delivered, implying TCP. We're worried about congesting the network if the logging protocol is too chatty, or dragging the apps to a crawl if it isn't responsive.

Some possibilities I've considered are:

• syslog (it seems perfect, but my boss has an animus against this so I may not be able to choose it).
• scribe from facebook (but it seems a bit heavyweight with a server on every machine - not every log message needs ultra-reliability).
• using a message queue like rabbitmq which can have multiple queues tuned to different levels of transaction safety.
• worst case, I can write my own from scratch.

Do you have other suggestions? What centralized logging solutions have you used, and how well did they work out?

Edit: I was leaning towards scribe, because its store-and-forward design decouples the running app from network latency. But after struggling to install it, I found that (1) it's not available as a binary package - nowadays that's unforgivable - and (2) it depends intimately on a library (thrift) that isn't available as a binary package either! And worst of all, it wouldn't even compile properly. That is not release quality code, even in open source.

Answer 1

You might also consider using SNMP alerts.

Answer 2

Hi, we have used successfully used the zeromq (www.zeromq.org) queue server for the logs of a distributed application scenario like the yours. It's very very reliable and incredibly fast and we looked at it after a not so successful implementation with Spread (http://www.spread.org). With zeromq a single server is able, for us, to handle more than 70 different logs from a medium to high busy distributed application. It receives the data from LAN and via the Internet. If you need a detailed queue server comparaison, look at this page:

http://wiki.secondlife.com/wiki/Message%5FQueue%5FEvaluation%5FNotes

hope it helps!

Answer 3

The other examples might be great, but I've had good luck with Syslog-NG. It is extremely flexible and configurable; although it's pretty easy to pick it up and do something useful with quickly.

Answer 4

Syslog is good if you intend to focus only on infrastructure logs (e.g. on system level). I heard that KIWI Syslog Server is a good one, though didn't try it myself. On the other hand, if you want to log an application related stuff, a syslog is perhaps not the best option for this. In case you use apache logging services (log4j, log4xxx and the rest), then logFaces would be a good solution as it's built particularly for aggregating multiple applications in one place. Works with both TCP or UDP connections and has decent log viewer and database integration.