Storing big int array partially in database , to reduce memory in java

Question

Hi Experts,

I have a class that has int[] members. The arrays grow very big, about 56M in size.(arrays are expandable with implementation similar to arraylist). Now I want to partially store the array in database, to improve memory. I have oracle database at my disposal. I would also like to cache more used indexes. I am wondering if there exists an api for this purpose. Otherwise please suggest an implementation approach.

Answer 1

I would look at BerkeleyDB rather than a relational DB. That will take care of writing to disk and caching.

Not sure whether it would make more sense to store the ints directly as values or in stripes--I'd start with the first and then see how performance looked before complicating matters with striping.

Answer 2

Why don't you serialize the array object. For Every array size of 2m , the class is serialized and stored in a file.finally deserialize the classes and merge the array.

Answer 3

56M isn't very big, unless you are developing for a mobile device. 1 Gb cost around $100 in a server/workstation so 56MB is worth about $6. If you have many GBs it may be worth doing this.

If you want an expandable int[] I suggest you look at TIntArrayList which is an ArrayList like collection which uses int[].

BTW: An RDBMS data will use far more memory than you are intending to save, and it is likely to 10x or more slower depending on what you do with the array.