tags:

views:

283

answers:

6

Below is a small test I've coded to educate myself on references API. I thought this would never throw OOME but it is throwing it. I am unable to figure out why. appreciate any help on this.

public static void main(String[] args)
{
    Map<WeakReference<Long>, WeakReference<Double>> weak = new HashMap<WeakReference<Long>, WeakReference<Double>>(500000, 1.0f);

    ReferenceQueue<Long> keyRefQ = new ReferenceQueue<Long>();
    ReferenceQueue<Double> valueRefQ = new ReferenceQueue<Double>();

    int totalClearedKeys = 0;
    int totalClearedValues = 0;

    for (long putCount = 0; putCount <= Long.MAX_VALUE; putCount += 100000)
    {
        weak(weak, keyRefQ, valueRefQ, 100000);

        totalClearedKeys += poll(keyRefQ);
        totalClearedValues += poll(valueRefQ);

        System.out.println("Total PUTs so far    = " + putCount);
        System.out.println("Total KEYs CLEARED so far    = " + totalClearedKeys);
        System.out.println("Total VALUESs CLEARED so far = " + totalClearedValues);
    }

}

public static void weak(Map<WeakReference<Long>, WeakReference<Double>> m, ReferenceQueue<Long> keyRefQ,
        ReferenceQueue<Double> valueRefQ, long limit)
{
    for (long i = 1; i <= limit; i++)
    {
        m.put(new WeakReference<Long>(new Long(i), keyRefQ), new WeakReference<Double>(new Double(i), valueRefQ));
        long heapFreeSize = Runtime.getRuntime().freeMemory();
        if (i % 100000 == 0)
        {
            System.out.println(i);
            System.out.println(heapFreeSize / 131072 + "MB");
            System.out.println();
        }

    }
}

private static int poll(ReferenceQueue<?> keyRefQ)
{
    Reference<?> poll = keyRefQ.poll();
    int i = 0;
    while (poll != null)
    {
        // 
        poll.clear();
        poll = keyRefQ.poll();
        i++;
    }
    return i;

}

}

And below is the log when ran with 64MB of heap

Total PUTs so far    = 0
Total KEYs CLEARED so far    = 77982
Total VALUESs CLEARED so far = 77980
100000
24MB

Total PUTs so far    = 100000
Total KEYs CLEARED so far    = 134616
Total VALUESs CLEARED so far = 134614
100000
53MB

Total PUTs so far    = 200000
Total KEYs CLEARED so far    = 221489
Total VALUESs CLEARED so far = 221488
100000
157MB

Total PUTs so far    = 300000
Total KEYs CLEARED so far    = 366966
Total VALUESs CLEARED so far = 366966
100000
77MB

Total PUTs so far    = 400000
Total KEYs CLEARED so far    = 366968
Total VALUESs CLEARED so far = 366967
100000
129MB

Total PUTs so far    = 500000
Total KEYs CLEARED so far    = 533883
Total VALUESs CLEARED so far = 533881
100000
50MB

Total PUTs so far    = 600000
Total KEYs CLEARED so far    = 533886
Total VALUESs CLEARED so far = 533883
100000
6MB

Total PUTs so far    = 700000
Total KEYs CLEARED so far    = 775763
Total VALUESs CLEARED so far = 775762
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at Referencestest.weak(Referencestest.java:38)
    at Referencestest.main(Referencestest.java:21)
+4  A: 

from http://weblogs.java.net/blog/2006/05/04/understanding-weak-references

I think your use of HashMap is likely to be the issue. You might want to use WeakHashMap

To solve the "widget serial number" problem above, the easiest thing to do is use the built-in WeakHashMap class. WeakHashMap works exactly like HashMap, except that the keys (not the values!) are referred to using weak references. If a WeakHashMap key becomes garbage, its entry is removed automatically. This avoids the pitfalls I described and requires no changes other than the switch from HashMap to a WeakHashMap. If you're following the standard convention of referring to your maps via the Map interface, no other code needs to even be aware of the change.

DaveC
There is also a potential problem that references are not enqueued immediately after being found by the GC.
Tom Hawtin - tackline
i've gone thru that article before coding this program...and i know this can be fixed using WeakHashMap...i am trying the understand and learn the reference api..so i posted the question here..tx anyway
Pangea
what is enquing?
Pangea
@Pangea - enqueue means "add to a queue". As any good web dictionary will tell you.
Stephen C
+3  A: 

The heart of the problem is probably that you're filling your heap with WeakReference-objects, the weak references are cleared when you're getting low on memory, but the reference objects themselves are not, so your hashmap is filling up with boat-load if WeakReference objects (not to mention the object array the hashmap uses, which will grow indefinitely), all pointing to null.

The solution, as already pointed out, is a weak hashmap, which will clear out those objects if they're no longer in use (this is done during put).

EDIT:
As Kevin pointed out, you already have your reference-queue logic worked out (I didn't pay close enough attention), a solution using your code is to just clear it out of the map at the point where the key has been collected. This is exactly how weak hash map works (where the poll is simply triggered on insert).

roe
I marked this up as it addresses the the problem, but the solution really is not a `WeakHashMap`. Since he has already coded all the poll logic, he should add code to remove the `WeakHashReference` objects from the `Map` that no longer reference any real object.
Kevin Brock
@Kevin Brock: You're very correct. I didn't pay enough attention. Thanks, I'll incorporate that.
roe
A: 

I'm not a java expert at all, but I know in .NET when doing a lot of large object memory allocation you can get heap fragmentation to the point where only small pieces of contiguous memory are available for allocation even though much more memory appears as "free".

A quick google search on "java heap fragmentation" brings up some seemingly relevant result although I haven't taken a good look at them.

Brian Hasden
Java can 'defragment' the heap if necessary, usually called 'compacting'.
roe
Does compacting actually happen when called or does it just notify the garbage collector that compacting needs to occur? Again, falling back to .NET when, for example, you try to force garbage collection it doesn't necessary force a collection to happen immediately. I understand garbage collection and compacting are different, just wondering if compacting follows the same sort of "I'll do compacting when I have the time" procedure.
Brian Hasden
its part of java's GC algorithm "Mark-Sweep-Compact"...so when a GC is invoked then the compacting also happens
Pangea
That's exactly my point. As a test, you might try putting a sleep for a few seconds in between the main loop iterations to see if that gives allows the garbage collector to kick off. Your application might be eating up resources and cpu cycles so quickly that the garbage collection doesn't run. It's just a thought but I have run into a very similar issue where the large object heap was becoming fragmented. You can see more about it at http://www.codeproject.com/KB/cs/Large_Objects____Trouble.aspx. Obviously that's .Net code but most of the concepts/issues are portable to Java.
Brian Hasden
@Brian - I can assure you that the current generation of Java heaps do not suffer from fragmentation problems. The HotSpot GC algorithms *will* compact the heap.
Stephen C
+1  A: 

Even when your weak references let go of the things they are referencing, they still do not get recycled themselves.

So eventually your hash will fill up with references to nothing and crash.

What you would need (if you wanted to do it this way) would be to have an event triggered by object deletion that went in and removed the reference from the hash. (which would cause threading issues you need to be aware of as well)

Bill K
tx for confirming on my understanding
Pangea
A: 

An other problem might be that Java for some reason don't always activate its garbadge collecter when running out of memmory, so you might need to insert explicit calls to activate the collector. Try something like

if( (putCount%1000)===0) Runtime.getRuntime().gc();

in your loop.

Edit: It seems that the new java implementations from sun now does call the garbadge collector before throwing OutOfMemmoryException, but I am pretty sure that the following program would throw OutOfMemmoryException with jre1.3 or 1.4

public class Test { public static void main(String args[]) { while(true) { byte []data=new byte[1000000]; } } }

Martin Tilsted
I thought the GC is always invoked if necessary before rather than terminating the JVM with OOM..is my understanding correct
Pangea
Calling gc doesn't necessarily force garbage collection. Most of the time it may, but it's not guaranteed.
Brian Hasden
the JVM will always attempt a GC if it's unable to allocate more memory. Triggering a GC is only useful if you're currently inactive and would like to use the spare cycles to clean up.
roe
A: 

Other's have correctly pointed out what the problem is; e.g. @roe, @Bill K.

But another way to solve this kind problem (apart from scratching your head, asking on SO, etc), is to look and see how the Sun recommended approach works. In this case, you can find it in the source code for the WeakHashMap class.

There are a few ways to find Java source code:

  • If you have a decent Java IDE to running, it should be able to show you the source code of any class in the class library.

  • Most J2SE JDK downloads include source JAR files for (at least) the public API classes.

  • You can specifically download full source distributions for the OpenJDK-based releases of Java.

But the ZERO EFFORT approach is to do a Google search, using the fully qualified name of the class with ".java.html" tacked on the end. For example, searching for "java.util.WeakHashMap.java.html" gives this link in the first page of search results.

And the source will tell you that the standard WeakHashMap implementation explicitly polls its reference queue to expunge stale (i.e. broken) weak references from the map's key set. In fact, it does this every time you access or update the map, or even just ask for its size.

Stephen C