Physical/in-Memory database. (for logging purposes on my website)

Question

We all know MYSQL. We all know memcached.

I love memcached. You have a string key , and it returns a value. Dead simple.

Is there a database for memory? For example, I am building a website that needs to keep track of LOGGING. Everything people do...I need to keep track of. But it would be slow to write to disk every time someone hits a page...

So I want to keep it in memory. The problem is, memcached is way too simple for this. I don't want to store this in memcached:

log1:query:apple,12-11-2009:32:33,3,5,2

Is there a tool as simple to use as memcached, but offers more functionality rather than basic key-value?

The key is simplicity to use, simplicity to learn, yet it works!

Answer 1

HSQLDB is the most prominent one I've seen. It's small, fast, uses standard SQL, and is used by OpenOffice and other projects.

Answer 2

I think SQLite may fit your needs. It is a full-fledged relational database (in fact, you control it using SQL commands, as the name implies), but very lightweight and allows databases to be maintained entirely in memory.

Answer 3

I would discourage you from using a different database for storing the logging info versus your business data. When you want to do real-life reporting - for example, how many users of group X did activity Y - you will want to do joins between the "business data" and the "logging data". If that data is in a different database management system, you'll have troubles.

Really, the tax for doing a single write to disk isn't that high (note that your HDD controller is smart enough to batch up the writes - the writes aren't blocking).

Answer 4

Using the memory engine under MySQL would surely keep it fast. The only problem is that it's not persistent, so if MySQL is restarted your data goes away (good for session data etc).

Either that or just append a file on the filesystem. If you need to merge logs between servers use facebook's scribe.

Answer 5

I have really been impressed with the capabilities of Flare. Its uses the memcached protocol for connections, and Tokyo Cabinet for persistence. The end result is that memcached clients 'just kinda work': http://labs.gree.jp/Top/OpenSource/Flare-en.html

I am not running this in production yet, btw.

Also, you get clustering with dynamic host addition and a number of replication options so you can do some funky stuff like circular replication sharded inherently by your memcached key hashing strategy! Certainly worth a look as it should be (hopefully!) a configuration property change to point to Flare instead of memcached.

Answer 6

You've already got plenty of caching/buffering from the database and the operating system. You'd be adding major complexity for no gain, especially for mostly a write-only activity. You might consider doing logging like everyone else - write to a text file.