string - java outOfMemoryError with stringbuilder

Question

I'm getting a java outOfMemoryError when I call this method - i'm using it in a loop to parse many large files in sequence. my guess is that result.toString() is not getting garbage collected properly during the loop. if so, how should i fix it?

private String matchHelper(String buffer, String regex, String method){
    Pattern abbrev_p = Pattern.compile(regex);//norms U.S.A., B.S., PH.D, PH.D.
    Matcher abbrev_matcher = abbrev_p.matcher(buffer);
    StringBuffer result = new StringBuffer();
    while (abbrev_matcher.find()){
            abbrev_matcher.appendReplacement(result, abbrevHelper(abbrev_matcher));
    }
    abbrev_matcher.appendTail(result);
    String tempResult = result.toString(); //ERROR OCCURS HERE
  return tempResult;

}

Answer 1

Written this way, you'll need roughly 6 bytes of memory for every character in the file.

Each character is two bytes. You have the raw input, the substituted output (in the buffer), and you are asking for a third copy when you run out of memory.

If the file is encoded in something like ASCII or ISO-8859-1 (a single-byte character encoding), that means it will be six times larger in memory than on disk.

You could allocate more memory to the process, but a better solution might be to process the input "streamwise"—read, scan, and write the data without loading it all into memory at once.