Is it better to use an in-memory database (e.g. SQLite) than to keep everything in HashMap or other data structures?

Question

I need to have very fast access to a big Map - several millions of entries. Is it worth using an SQLite in-memory database to keep that map as opposed to just having that HashMap in memory?

Answer 1

its depends on the services you require from your data structure. do you only need to retrieve values from the map? or do you need to do a complex query or sorting?.

there is nothing magical about a database internal structure, to make it arbitrary faster then a simple data structure . In the database there are more facilities to manipulate large sets of data that probably will cost in an overheard of CPU and memory. if you only need a dictionary like functionality , go with a map, for something more complex consider a database

Answer 2

It depends, but if your data model is simple enough to fit into a Map, and you don't need to save the data between runs of your program, then an in-memory database will most likely be overkill. Databases are designed for more complex data models, safe parallel access and update using transactions, complex queries, constraints, and so on.

If you decide that a Map is appropriate, then you should carefully select the kind of Map you need. Take a look at the full range of Maps available in java.util.collections, and also look hard at Google Collections, which extends Java's in some very nice ways. Also look at java.util.concurrent, which has a good ConcurrentHashMap that will permit your data structure to be used concurrently by multiple threads.

Be sure to consider how you construct your Map. If you use a HashMap, setting the initial capacity and load factor at construction time may have some impact on performance.

Another thing to do here is hide your implementation behind a fascade class you write. That way if you decide to switch approaches you won't impact your client code.

Answer 3

I think that there is nothing faster than a hashmap in memory if you only query by key.

Answer 4

In addition to Alon and Jim's excellent remarks, I would suggest trying both approaches and benchmarking the performance. Besides being fun (in an admittedly geeky way), this test will force you to encapsulate your data structure in just the right way, so that only the essential functionality is exposed.