Java: Garbage Collection

Question

I'm having an out of memory error. I have a large range of inputs (2^40), that is too large to hold at once. Each input is a String[].

Instead, I thought I'd run my test program on each input, write the results to a file, then discard the input. The length of the longest input is 42, so that isn't an error causing the overflow. I don't think I understand garbage collection.

PowerSet is like a lazy list - the result isn't calculated until .next() is called. Each result returns one subset of baseSet. baseSet is a String[] with length 40. runTests does some analysis on the input and writes it to a file.

PowerSet argSetSet = powerset(baseSet); 
while (argSetSet.hasNext()) {
    runTests(argSetSet.next()); //saves output to file
}

This causes an out of memory error. But I'm not saving the result of argSetSet.next() anywhere, so why should this occur? I don't need to store any data from next() or runTests(). How can I make sure it's all being garbage collected?

If I comment out runTests(), it runs fine.

Answer 1

There isn't enough code to understand what is going on, primarily PowerSet, but PowerSet has to calculate the String array to return the next method. It could be that it is holding on to that object.

The memory issue is either in the runTests method or the PowerSet class. It isn't in the code you posted.

Answer 2

What's in baseSet? I'm guessing that's using up a ton of memory. This is probably exacerbated when PowerSet uses baseSet internally.

Answer 3

The fact that you aren't storing the results of .next() anywhere isn't relevant, what is relevant is what .next() is actually doing.

Are you setting the heap size to a non-default size? What settings are you using to start the JVM? The default heap size of the JVM is only 64M, so one trillion entries certainly cannot fit into that space.

Answer 4

Attach a profiler like jvisualvm and investigate where your memory goes. You might be surprised :)

Answer 5

[Clippy-like icon] It appears you are calculating the powerset of a very large set. Do you want to increase your heap size?

What concerns me is that you're saying that this is a lazy list, which means that the entire powerset isn't actually in memory, but only a piece of it is in memory when you call .next(). HOWEVER, depending on what .next() actually returns (the size of the array) then the default heap size is most likely not enough.

You can increase the size of the heap by specifying -Xmx1024m (setting the maximum value size of the heap to 1gb). Obviously you can tweak that size, but this will allow you to test to see if it scales. This is not an end-all solution, but it should at least give you some runway.