What's a good way to collect logs from Amazon EC2 instances?

Question

My app is hosted on an Amazon EC2 cluster. Each instance writes events to log files. I need to collect (and data mine) over these logs at the end of each day. What's a recommended way to collect these logs in a central location? I have thought of several options, not sure which way to go:

1. scp them to an instance using a cron job
2. Log all events over TCP/IP to an instance

Answer 1

I haven't implemented it yet, but I came across Facebook Scribe, and it seems like a good idea. http://developers.facebook.com/scribe/

Until I get that going I've been doing just what you mention for #1 - I have a cron job which uses sftp to pull the files. I chose this because even if I did #2, I've had instances where the ec2 machine crashes and I ended up having to pull the logfiles off anyway.

Answer 2

Have a look at the free version of Splunk - it will handle the remote log file collection and also give you some really nice search and analysis tools.

Answer 3

Use syslog-ng, a popular tool that transfers log messages over TCP, optionally encrypted.

http://www.balabit.com/network-security/syslog-ng/