couchdb hovercraft limitations, storing arbitrary erlang terms into Couchdb

Question

so Ive been messing with hovercraft and ran into some anoying limitation, that are probably there due to the fact that internally couchdb deals with key/value pairs associated with a document as opaque strings (json strings).

namelly:
- doc _id's can only be binary strings (utf8) - no complex erlang terms allowed here
- key/value pairs can only be binatry_strings or atoms or lists (no tuples, or arbitrary binaries allowed).

I was looking forward to storing arbitrary erlang terms in there, without encoding them as JSON first. yes this is possible, but then the entire view system (and the http api,notifications,verification,indexing) just stops working.
that too is fine, I could code around it, not use futon, map/reduce over documents manually and store results as documents (which actually is better since then those results can be replicated to other DBs/nodes, unlike views results(which dont replicate - correct me if Im wrong)).

the real problem seems that without views one cannot get a list of all the keys that are stored in a db, at least not via the current hovercraft api. that is a show stoper for mapreducing manually over an entire db, without knowing prior what the doc _id's are.

any ideas as to how I can get a list of these keys in a db? via erlang calls, possibly into the internals of couchdb?

its even more obvious to me now that the direct erlang api for couchdb was a total afterthough.

Answer 1

As the author of Hovercraft, I agree with the statement "the direct erlang api for couchdb was a total afterthough."

You should only use Hovercraft if you are converting CouchDB from an HTTP server to say, an SMTP server. HTTP will scale much better than Hovercraft.

It should be possible to use the internal _changes API to iterate over all the docs in the database and maintain a secondary index incrementally.

As for storing non-JSON data in CouchDB, that sounds risky as no one will be looking out to make sure we don't break your use case.

But if you are having fun, by all means, continue. And I love getting patches to Hovercraft, so any little thing will probably get rolled back in.

Thanks, Chris