Hi all,
While writing huge data to a file using java, how can one avoid java heap memory out of space error from occuring?..This is causing some failures for my program..
Regards,
magggi
A:
Use the -Xmx parameter when launching the java program to set the maximal heap size:
java -Xmx256M com.foo.MyClass
gawi
2010-09-09 18:53:08
If the file is that huge, allocating more memory isn't the best way to deal with it.
Colin Hebert
2010-09-09 18:54:57
Note that the initial statement was rather vague. I tend to give short and simple answers when the context is not well defined. It was not even mentioned that he could modify the program. I've observed that StackOverflowers will always prefer the "cleanest" solution (disregarding the complexity) over simple, straight-forward, but non-scalable/slow/occasionally-incorrect solution. When the context of the question is not well-defined, maybe I should always give 2 answers: the hack and the good-practice.
gawi
2010-09-10 00:49:03
A:
Did you try passing the memory parameters to the applicaiton when starting it?
For example the following sets the maximum memory to 1 GB
java -Xms512m -Xmx1024m myClass
Kushal Paudyal
2010-09-09 18:55:54
+2
A:
Your problem is that you're using a single StringBuffer to handle all your content. You should try to flush your data more often. You will gain performance and memory.
Without the original code I can't help you more, but try to flush your buffer from time to time.
Not advised in your case
If you still want to use your StringBuffer and flush it only once, then you'll have to increase your heap space with the -Xmx***M option on the java command.
Resources :
- oracle.com - basic IO
- oracle.com - Java command line tool (see -Xmx part for heap)
On the same topic :
Colin Hebert
2010-09-09 18:56:45
A:
A couple of options
Increase Java Heap Size using the -Xmx JVM parameter. After -Xmx, write the number of megabytes your program needs to successfully run.
Use Streaming. Compute the next N bytes in the file and then write them using your FileOutputStream, throwing them away before computing the next N bytes. More code would be necessary to know if and how this would work for your program.
Edit: From what you are saying, it sounds like you could wrap the FileOutputStream in a BufferedOutputStream, drop the StringBuffer entirely and just write directly to the BufferedOutputStream.
ILMTitan
2010-09-09 18:58:11
+1
A:
From your comments it sounds like you're iterating over some objects that fill up a StringBuffer and, once you're done iterating, you flush the entire StringBuffer to a file. Instead of doing this, you can either write directly to an output stream on each iteration (recommended) or, on each iteration, write to your StringBuffer, flush it to the output stream, and then clear the StringBuffer for the next iteration (not preferred). This should lower your memory usage.
I don't recommend increasing the heap size -- it doesn't sound like the right solution to this problem.
dlongley
2010-09-09 19:21:00