Efficient persistent storage for simple id to table of values map for java

Question

I need to store some data that follows the simple pattern of mapping an "id" to a full table (with multiple rows) of several columns (i.e. some integer values [u, v, w]). The size of one of these tables would be a couple of KB. Basically what I need is to store a persistent cache of some intermediary results.

This could quite easily be implemented as simple sql, but there's a couple of problems, namely I need to compress the size of this structure on disk as much as possible. (because of amount of values I'm storing) Also, it's not transactional, I just need to write once and simply read the contents of the entire table, so a relational DB isn't actually a very good fit.

I was wondering if anyone had any good suggestions? For some reason I can't seem to come up with something decent atm. Especially something with an API in java would be nice.

Answer 1

Have you looked at Berkeley DB? That sounds like it may fit the bill.

---

Edit:

I forgot to add you can gzip the values themselves before you store them. Then just unzip them when you retrieve them.

Answer 2

This sounds like a job for.... new ObjectOutputStream(new FileOutputStream(STORAGE_DIR + "/" + key + ".dat"); !!

Seriously - the simplest method is to just create a file for each data table that you want to store, serialize the data into and look it up using the key as the filename when you want to read.

On a decent file system writes can be made atomic (by writing to a temp file and then renaming the file); read/write speed is measured in 10s of MBit/second; look ups can be made very efficient by creating a simple directory tree like STORAGE_DIR + "/" + key.substring(0,2) + "/" + key.substring(0,4) + "/" + key which should be still efficient with millions of entries and even more efficient if your file system uses indexed directories; lastly its trivial to implement a memory-backed LRU cache on top of this for even faster retrievals.

Regarding compression - you can use Jakarta's commons-compress to affect a gzip or even bzip2 compression to the data before you store it. But this is an optimization problem and depending on your application and available disk space you may be better off investing the CPU cycles elsewhere.

Here is a sample implementation that I made: http://geek.co.il/articles/geek-storage.zip. It uses a simple interface (which is far from being clean - its just a demonstration of the concept) that offers methods for storing and retrieving objects from a cache with a set maximum size. A cache miss is transfered to a user implementation for handling, and the cache will periodically check that it doesn't exceed the storage requirements and will remove old data.

I also included a MySQL backed implementation for completion and a benchmark to compare the disk based and MySQL based implementations. On my home machine (an old Athlon 64) the disk benchmark scores better then twice as fast as the MySQL implementation in the enclosed benchmark (9.01 seconds vs. 18.17 seconds). Even though the DB implementation can probably tweaked for slightly better performance, I believe it demonstrates the problem well enough.

Feel free to use this as you see fit.

Answer 3

I'd use EHCache, it's used by Hibernate and other JEE libraries, and is really simple and efficient:

To add a table:

List<List<Integer>> myTable = new(...)
cache.put(new Element("myId", myTable));

To read:

List<List<Integer>> myTable = (List<List<Integer>>) cache.get("myId").getObjectValue();

Answer 4

Apache Derby might be a good fit if you want something embedded (not a separate server).

There is a list of other options at Lightweight Data Bases in Java

Answer 5

It seems that Key=>Value Databases are the thing you search for.

Maybe SuperCSV is the best framework for you!

If you don't want to use a relational database, you can use JAXB to store your Objects as XML files!

There is also a way with other libraries like XStream

If you prefer XML, then use JAXB or XStream. Otherwise you should have a look at CSV libraries such as SuperCSV. People who can life with serialized java files can use the default persistance mechanism like Guss said. Direct Java persistance may be the fastest way.

Answer 6

You can use JOAFIP http://joafip.sourceforge.net/ It make you able to put all your data model in file and you can access to it, update it, without reloading all in memory.

Answer 7

If you have a couple of KB, I don't understand why you need to "compress the size of this structure on disk as much as possible" Given that 181 MB of disk space costs 1 cent, I would suggest that anything less than this isn't worth spending too much time worrying about.

However to answer your question you can compress the file as you write it. As well as ObjectOutputStream, you can use XMLExcoder to serialize your map. This will be more compact than just using ObjectOutputStream and if you decompress the file you will be able to read or edit the data.

XMLEncoder xe = new XMLEncoder(
    new GZIPOutputStream(
        new FileOutputStream(filename+".xml.gz")));
xe.writeObject(map);
xe.close();

Answer 8

Well, this is not Java, but a quick solution < 10 minutes: y_serial.py module :: warehouse Python objects with SQLite

"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."

http://yserial.sourceforge.net

Compression is seamlessly integrated into yserial which is nice because that dramatically reduces the size of BLOBs.

(Kindly tell us if the module works under Jython -- thanks ;-)