Java Fast Data Storage & Retrieval

Question

I need to store records into a persistant storage and retrieve it on demand. The requirement is as follows -

1. Extremely fast retrieval and insertion
2. Each record will have a unique key. This key will be used to retrieve the record
3. The data stored should be persistent i.e. should be available upon JVM restart
4. A separate process would move stale records to RDBMS once a day

What do you guys think? I cannot use standard database becuase of latency issues. In memory databases like HSQLDB / H2 has performace contraints. Moreover the records are simple string objects and do not qualify for SQL. I am thinking of some kind of flat file based solution. Any ideas? Any open source project? I am sure, there must be someone who has solved this problem before :)

Answer 1

MySQL with shards may be a good idea. However, it depends on what is the data volume, transactions per second and latency you need.

In memory databases are also a good idea. In fact MySQL provides memory-based tables as well.

Answer 2

Have you actually proved that using an out-of-process SQL database like MySQL or SQL Server is too slow, or is this an assumption?

You could use a SQL database approach in conjunction with an in-memory cache to ensure that retrievals do not hit the database at all. Despite the fact that the records are plaintext I would still advise using SQL over a flat file solution (e.g. using a text column in your table schema) as the RDBMS will perform optimisations that a file system cannot (e.g. caching recently accessed pages, etc).

However, without more information about your access patterns, expected throughput, etc. I can't provide much more in the way of suggestions.

Answer 3

Will all the records and keys you need fit in memory at once? If so, you could just use a HashMap<String,String>, since it's Serializable.

Answer 4

How much does it matter if you lose a record or two? Where are they coming from? Do you have a transactional relationship with the source?

If you have serious reliability requirements then I think you may need to be prepared to pay some DB Overhead.

Perhaps you could separate the persistence problem from the in-memory problem. Use a pup-sub approach. One subscriber look after in-memory, the other persisting the data ready for subsequent startup?

Distributed cahcing products such as WebSphere eXtreme Scale (no JEE dependency) might be relevent if you can buy rather than build.

Answer 5

Have a look at LinkedIn's Voldemort.

Answer 6

How bad would it be if you lose a couple of entries in case of a crash?

If it isn't that bad the following approach might work for you:

Create flat files for each entry, name of file equals id. Possible one file for a not so big number of consecutive entries.

Make sure your controller has a good cache and/or use one of the existing caches implemented in Java.

Talk to a file system expert how to make this really fast

It is simple and it might be fast. Of course you lose transactions including the ACID principles.

Answer 7

If all the data fits in memory, MySQL can run in memory instead of from disk (MySQL Cluster, Hybrid Storage). It can then handle storing itself to disk for you.

Answer 8

If you are looking for a simple key-value store and don't need complex sql querying, Berkeley DB might be worth a look.

Another alternative is Tokyo Cabinet, a modern DBM implementation.

Answer 9

Would a Tuple space / JavaSpace work? Also check out other enterprise data fabrics like Oracle Coherence and Gemstone.

Answer 10

What about something like CouchDB?

Answer 11

Sub millisecond r/w means you cannot depend on disk, and you have to be careful about network latency. Just forget about standard SQL based solutions, main-memory or not. In a ms, you cannot get more than 100 KByte over a GBit network. Ask a telecom engineer, they are used to solving these kind of problems.