Which key value store is the most promising/stable?

Question

I'm looking to start using a key/value store for some side projects (mostly as a learning experience), but so many have popped up in the recent past that I've got no idea where to begin. Just listing from memory, I can think of:

1. CouchDB
2. MongoDB
3. Riak
4. Redis
5. Tokyo Cabinet
6. Berkeley DB
7. Cassandra
8. MemcacheDB

And I'm sure that there are more out there that have slipped through my search efforts. With all the information out there, it's hard to find solid comparisons between all of the competitors. My criteria and questions are:

1. (Most Important) Which do you recommend, and why?
2. Which one is the fastest?
3. Which one is the most stable?
4. Which one is the easiest to set up and install?
5. Which ones have bindings for Python and/or Ruby?

Edit:
So far it looks like Redis is the best solution, but that's only because I've gotten one solid response (from ardsrk). I'm looking for more answers like his, because they point me in the direction of useful, quantitative information. Which Key-Value store do you use, and why?

Edit 2:
If anyone has experience with CouchDB, Riak, or MongoDB, I'd love to hear your experiences with them (and even more so if you can offer a comparative analysis of several of them)

Answer 1

They all have different features. And don't forget Voldermort (http://project-voldemort.com/) which is actually used/tested by LinkedIn in their production before each release.

It's hard to compare. You have to ask yourself what you need: e.g. do you want partitioning? if so then some of them, like CouchDB, won't support it. Do you want erasure coding? Then most of them don't have that. Etc.

Berkeley DB is a very basic, low level storage engine, that perhaps can be excused from this discussion. Several key-value systems are built on top of it, to provide additional features like replication, versioning, coding, etc.

Also, what does your application need? Several of the solutions contain complexity that may not be necessary. E.g. if you just store static data that won't change, you can store them under data's SHA-1 content hash (i.e. use the content-hash as key). In this case, you don't have to worry about freshness, synchronization, versioning, and lots of complexities can be removed.

Answer 2

Which do you recommend, and why?

I recommend Redis. Why? Continue reading!!

Which one is the fastest?

I can't say whether its the fastest. But Redis is fast. Its fast because it holds all the data in RAM. Recently, virtual memory feature was added but still all the keys stay in main memory with only rarely used values being swapped to disk.

Which one is the most stable?

Again, since I have no direct experience with the other key-value stores I can't compare. However, Redis is being used in production by many web applications like github and superfeedr among many others.

Which one is the easiest to set up and install?

Redis is fairly easy to setup. Grab the source and on a Linux box run make install. This yields redis-server binary that you could put it on your path and start it.

redis-server binds to localhost:6379 by default. Have a look at redis.conf that comes with the source for more configuration and setup options.

Which ones have bindings for Python and/or Ruby?

Redis has excellent Ruby and Python support. Ruby client is more feature complete ( for instance, it supports consistent hashing that the python client doesn't ) and popular ( redis-rb has 300+ watchers while redis-py has only 100+ ) compared to the python client.

[EDIT: response to xorlev's comment]

@xorlev: Thanks for the comment as I had missed something important to mention and you reminded me of it.

Memcached is just a simple key-value store. Redis holds everything in memory plus its also persistent. It writes back to disk in a non-blocking way. Redis also supports complex value types ( just look at slide number 2 ) like lists, sets and sorted sets and at the same time provides a simple interface to these value types.

There is also make 32bit that makes all pointers only 32-bits in size even on 64 bit machines. This saves considerable memory on machines with less than 4GB of RAM.

Redis is also very well documented and you can give it a try online.

Hope this helps.

Answer 3

I really like memcached personally.

I use it on a couple of my sites and it's simple, fast, and easy. It really was just incredibly simple to use, the API is easy to use. It doesn't store anything on disk, thus the name memcached, so it's out if you're looking for a persistent storage engine.

Python has python-memcached.

I haven't used the Ruby client, but a quick Google search reveals RMemCache

If you just need a caching engine, memcached is the way to go. It's developed, it's stable, and it's bleedin' fast. There's a reason LiveJournal made it and Facebook develops it. It's in use at some of the largest sites out there to great effect. It scales extremely well.

Answer 4

At this year's PyCon, Jeremy Edberg of Reddit gave a talk:

http://pycon.blip.tv/file/3257303/

He said that Reddit uses PostGres as a key-value store, presumably with a simple 2-column table; according to his talk it had benchmarked faster than any other key-value store they had tried. And, of course, it's very mature.

Ultimately, OverClocked is right; your use case determines the best store. But RDMBSs have long been (ab)used as key-value stores, and they can be very fast, too.

Answer 5

Just to make the list complete: there's Dreamcache, too. It's compatible with Memcached (in terms of protocol, so you can use any client library written for Memcached), it's just faster.

Answer 6

I've been playing with MongoDB and it has one thing that makes it perfect for my application, the ability to store complex Maps/Lists in the database directly. I have a large Map where each value is a list and I don't have to do anything special just to write and retrieve that without knowing all the different keys and list values. I don't know much about the other options but the speed and that ability make Mongo perfect for my application. Plus the Java driver is very simple to use.

Answer 7

There is also zodb.

Answer 8

One distinction you have to make is what will you use the DB for? Don't jump on board just because it's trendy. Do you need a key value store? or do you need a document based store? What is your memory footprint requirement? running it on a small VM or a separate one?

I recommend listing your requirements first and then seeing which ones overlap with your requirements.

With that said, I have used CouchDB/MongoDB and prefer to use MongoDB for its ease of setup and best transition from mysql style queries. I chose mongodb over sql because of dynamic schemas(no migration files!) and better data modeling(arrays, hashes). I did not evaluate based on scalability.

MongoMapper is a great MongoDB orm mapper for Ruby and there's already a working Rails 3 fork.

I listed some more details about why I prefered mongodb in my scribd slides http://tommy.chheng.com/index.php/2010/02/mongodb-for-natural-development/

Answer 9

I notice how everyone is confusing memcached with memcachedb. They are two different systems. The op asked about memcachedb.

memcached is memory storage. memcachedb uses Berkeley DB as its datastore.

Answer 10

I only have experience with Berkeley DB, so I'll mention what I like about it.

• It is fast
• It is very mature and stable
• It has outstanding documentation
• It has C,C++,Java & C# bindings out of the box. Other language bindings are available. I believe Python comes with bindings as part of its "batteries".

The only downside I've run into is that the C# bindings are new and don't seem to support every feature.

Answer 11

You need to understand what modern NoSQL phenomenon is about.
It is not about key-value storages. They've been available for decades (BerkeleyDB for example). Why all the fuss now ?
It is not about fancy document or object oriented schemas and overcoming "impedance mismatch". Proponents of these features have been touting them for years and they got nowhere.
It is simply about adressing 3 technical problems: automatic (for maintainers) and transparent (for application developers) failover, sharding and replication. Thus you should ignore any trendy products that do not deliver on this front. These include Redis, MongoDB, CouchDB etc. And concentrate on truly distributed solutions like cassandra, riak etc.

Otherwise you'll loose all the good stuff sql gives you (adhoc queries, Crystal Reports for your boss, third party tools and libraries) and get nothing in return.

Answer 12

Cassandra seems to be popular.

Cassandra is in use at Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick, Cisco, SimpleGeo, Ooyala, OpenX, and more companies that have large, active data sets. The largest production cluster has over 100 TB of data in over 150 machines.