java outOfMemoryError with stringbuilder

Question

I'm getting a java outOfMemoryError when I call this method - i'm using it in a loop to parse many large files in sequence. my guess is that result.toString() is not getting garbage collected properly during the loop. if so, how should i fix it?

private String matchHelper(String buffer, String regex, String method){
    Pattern abbrev_p = Pattern.compile(regex);//norms U.S.A., B.S., PH.D, PH.D.
    Matcher abbrev_matcher = abbrev_p.matcher(buffer);
    StringBuffer result = new StringBuffer();
    while (abbrev_matcher.find()){
            abbrev_matcher.appendReplacement(result, abbrevHelper(abbrev_matcher));
    }
    abbrev_matcher.appendTail(result);
    String tempResult = result.toString(); //ERROR OCCURS HERE
  return tempResult;

}

Answer 1

You could try returning a StringBuffer and setting it to null after use.

Answer 2

If your files to be processed are all very large, say more than a few hundred MB, then you really should go with stream processing instead of this "loading all into memory" way, just as @erickson suggested.

Otherwise, there are a few things you could try, all to reduce memory usage as much as possible:

1. Try properly enlarge your heap size if not yet (when applicable).
2. Give StringBuffer an initial size same as the lenght of the given String buffer. This should reduce the unnecessary memory usage while expanding the StringBuffer in the process. I assume it is only replacing certain words of the original string and should be more or less the same in length.
3. If possible, maybe you could return the generated StringBuffer object instead. Calling its toString() only after you get rid of the original String object.

Answer 3

I agree with the other responses ... but ... simply because the exception occurs there doesn't necessarily mean it's the problem. You may very well be leaking memory elsewhere and that just happens to be the place that it's revealed. You should run a profiler to examine memory usage and verify exactly what objects aren't being collected.

Answer 4

I reckon the problem with StringBuilder.append(). When Matcher appends sequence of characters to the Builder.

As explained in article about OutOfMemoryError with StringBuilder/StringBuffer, it is a known issue that append() will double the capacity if internal buffer chars if the capacity is not sufficient. Go for streams as suggested by Erickson.

Answer 5

Yes! Don't buffer in memory otherwise you'll run out of it specially if you're going over 2MB on I/O.

Recommended link for fixing and appending text: http://java.ittoolbox.com/documents/appending-data-to-a-file-18786