MongoDB or CouchDB or something else?

Question

I know this is another question on this topic but I am a complete beginner in the NoSQL world so I would love some advice. People at SO told me MySQL might be a bad idea for this dataset so I'm asking this. I have lots of data in the following format:

TYPE 1

ID1: String String String ...
ID2: String String String ...
ID3: String String String ...
ID4: String String String ...

which I am hoping to convert into something like this:

TYPE 2

ID1: String
ID1: String
ID1: String
ID1: String
ID2: String
ID2: String

This is the most inefficient way but I need to be able to search by both the key and the value. For instance, my queries would look like this:

• I might need to know what all strings a given ID contains and then intersect the list with another list obtained for a different ID.
• I might need to know what all IDs contain a given string

I would love to achieve this without transforming Type 1 into Type 2 because of the sheer space requirements but would like to know if either MongoDB or CouchDB or something else (someone suggested NoSQL so started Googling and found these two are very popular) would help me out in this situation. I can a 14 node cluster I can leverage but would love some advice on which one is the right database for this usecase. Any suggestions?

A few extra things:

• The input will mostly be static. I will create new data but will not modify any of the existing data.
• The ID is 40 bytes in length whereas the strings are about 20 bytes

Answer 1

MongoDB will let you store this data efficiently in Type 1. Depending on your use it will look like one these (data is in JSON):

Array of Strings

{ "_id" : 1, "strings" : ["a", "b", "c", "d", "e"] }

Set of KV Strings

{ "_id" : 1, "s1" : "a", "s2" : "b", "s3" : "c", "s4" : "d", "s5" : "e" }

Based on your queries, I would probably use the Array of Strings method. Here's why:

I might need to know what all strings a given ID contains and then intersect the list with another list obtained for a different ID.

This is easy, you get one Key Value look-up for the ID. In code, it would look something like this:

db.my_collection.find({ "_id" : 1});

I might need to know what all IDs contain a given string

Similarly easy:

db.my_collection.find({ "strings" : "my_string" })

Yeah it's that easy. I know that "strings" is technically an array, but MongoDB will recognize the item as an array and will loop through to find the value. Docs for this are here.

As a bonus, you can index the "strings" field and you will get an index on the array. So the find above will actually perform relatively fast (with the obvious trade-off that the index will be very large).

In terms of scaling a 14-node cluster may almost be overkill. However, Mongo does support auto-sharding and replication sets. They even work together, here's a blog post from a 10gen member to get you started (10gen makes Mongo).