Eventually consistent mnesia database with erlang. Best practices anyone?

Question

Hi.

I'm writing a bittorrent tracker in erlang. Given the nature of the service, I won't need absolute consistency (ie. a client can be perfectly happy with a slightly outdated list of peers or torrent status).

My strategy so far has been to create mnesia tables in RAM with disc_copies enabled, so to have mnesia automatically dump the memory to disk when the log size exceeds a certain size.

If the server crashes, some information will be lost. Not a big deal.

A different approach would be to instance two tables (one ram only and one disk only) and have a process copy from ram to disk every minute or so. This is more naive, but would allow to dump just a subset of what's in memory, reducing the overall disk overhead and possibly avoid the usage of a log altogether (I'm actually not sure about this last statement).

I'm sure there are many other ways to do this. What's yours?

-teo

Answer 1

Here is a project that might interest you http://github.com/jlouis/etorrent/tree/master.

Answer 2

You may want to checkout redis & erldis. Redis takes the second approach: everything is stored in memory then periodically dumped to disk.

Answer 3

This is certainly off-topic in respect to your original question, but if you're really writing a pure tracker, then it might be best to forego persistence completely and keep the data solely in memory.

For a minimal tracker, an announce weighs in at only a few bytes: 16 bytes for the SHA1-hash, 6 bytes for the peer's IP and port, and some more bytes as you'll need to keep a timestamp as well. But even with a bit of overhead, you'll be able to literally keep millions of records in memory.