Reccomend an indexed file format that can be updated via random access in Java

Question

I need an indexed file format that can hold a few hundred large variable sized binary blobs.

Blobs are around 1-5MB and the file could be as large as 1 GB. I need to be able to quickly find, read, add and remove blobs without recreating the the entire file. I have no need to compress the blobs, however if blobs were removed, I'd like to reclaim or reuse the space.

Ideally there would be a Java API.

I'm currently doing this with a ZIP format, but there's no known way to update a ZIP file without recreating it and performance is bad.

I've looked into SQLite but its blob performance was slow, and its overkill for my needs.

Any thoughts, or should I roll my own? And if I do roll my own, any book or web page suggestions?

Thanks...

Answer 1

Berkeley DB Java Edition does what you need. It's free.

Answer 2

You need some virtual file system. Our SolFS is the one of the options yet we have only JNI layer, as the engine is written in C. There exists one more option, CodeBase, but as they don't provide an evaluation version of their file system, I know a few about it.

SolFS is ideally suitable for your task, because it lets you have alternative streams for files and associate searchable metadata with each file or even alternative stream.