tags:

views:

31

answers:

3

We keep most of our logs in a dedicated database table. We have written custom appenders for log4j and log4net, have a fixed log schema with lots of handy columns, and are quite happy with it.

Is that the "best practice" (for sites smaller in scale than Facebook, where a simple DB table just won't scale)?

+1  A: 

Assuming your running a Linux-based webserver, write your logs to a simple log file, and have a cron job bzip the file daily. You can bzcat the file to get at it's contents. The same cron script could remove files older than a given threshold, or you could remove old logs manually. This is the pretty accepted scheme that most daemons use, directly or indirectly via syslog.

Just remember to write the log file outside of the publicly accessible web root, or people can guess your log file names and download potentially revealing data.

meagar
Why do you think this is preferable to a database? It's less flexible. Have you ever tried storing them in a DB or DB-like system? (see my comment to @shoosh)
ripper234
A: 

Database logs are not compressed or moved to different boxes easily. On the other hand, searching logs on a database probably has an advantage.

Are you worried about your current solution? If you do not have problems with it, I wouldn't worry.

alex
I'm not worried - our solution works very well for us. I just wanted to know what everyone else was doing.
ripper234
A: 

Why are you using a database? Do you often find yourself making complex queries on it? Or can the queries you're using be translated to simple grep searches?
That's the main question you should ask yourself when choosing between a simple textual log file and DB table.

shoosh
Yes, we find keeping them in a database invaluable.We can do selects against a date range, component, log level, etc... I don't see how to do this with grep.Some other columns in our schema are PID/ThreadID (useful for multithreaded debugging). We have several others columns I just can't remember a.t.m.
ripper234