What kind of database would be best suited to maintaining a relatively large list of items with a counter for each item that is updated often in real time?

Question

Let's pretend it's for word frequency counts in a web crawler. Is relational the way to go (I'm imagining a simple two-column table) or is there a NoSQL option better suited to this task?

When I say better, I mean more conceptually suited to the task. I'm not really concerned with scalability, just simplicity and an obvious conceptual mapping to the task at hand. In the way that, for me at least, CouchDB maps much more sensibly to a blog than MySQL does.

Answer 1

first - are you looking for paid database or a free one ? if you have 1 or 2 tables maybe 1 or 2 indexes, no need for enterprise features (clustering/replication/Flashbacks) then any DB can do good from mysql (the free one) to sql express(still free) , sql server and oracle which both are commercial and cost money.

you need to understand that the hardware might have a role here too, as the schema looks very simple and there wouldn't be much optimization possible - but then again - I don't know your exact needs...

(on the other hand if you're talking about extreme large tables with a lot of read and writes - and I mean A LOT you might need a configuration of 2 active nodes and other advanced features than might push you towards paid databases)

Answer 2

Have a look at the papers and ideas behind Google BigTable (and the MapReduce operations possible on it). There's other implementations that think in that box; you're really implementing a distributed hash table, to give you some juice to throw at Google.

Answer 3

If this is a thing that you'll only run on one machine I'd just use an internal datastructure, a red-black tree or perhaps a trie for something as simple and small as this.. Or I'd embed a key/value pair database such as BerkeleyDB.