key - > value store with binary attachments

Question

An extra requirement is that the attachments can be stored as a stream, as there might be potentially very large binaries that has to be saved. Videos etc.

I have looked at Voldemort and other key value stores, but they all seem to expect byte arrays, which is completely out of the question.

This should, preferrably, be written in Java, and be embeddable.

The use case is: I have written a HTTP Cache library which has multiple backends.

I have a Memory based one (using hashmap and Byte arrays), Derby database, persistent hashmap with file attachment, EHCache with file attachment.

I was hoping there was something out there which didn't use the file system, or if it does, it's transparent from the API.

I am storing the Headers with some more meta information in a datastore. But I also need to store the payload of the HTTP response.

The HTTP response payload might be VERY big, thats why I need to use streaming.

Answer 1

Why is a byte[] value out of the question? Any object graph can be serialized into a byte array!

Have you looked at sleepycat's Berkeley DB (it's free)?

EDIT - having seen jhedding's comment, it seems like you need to store data which is too big to fit into a single JVM in one go. Have you:

1. Checked that it won't fot into a 64-bit JVM?
2. Tried using a network file system? (NAS or whatever)