To Compress a big file in a ZIP with Java

Question

I have the need to compress a one Big file (~450 Mbyte) through the Java class ZipOutputStream. This big dimension causes a problem of "OutOfMemory" error of my JVM Heap Space. This happens because the "zos.write(...)" method stores ALL the file content to compress in an internal byte array before compressing it.

      origin = new BufferedInputStream(fi, BUFFER);
  ZipEntry entry = new ZipEntry(filePath);
  zos.putNextEntry(entry);

  int count;
  while ((count = origin.read(data, 0, BUFFER)) != -1)
  {
   zos.write(data, 0, count);
  }
  origin.close();

The natural solution will be to enlarge the heap memory space of the JVM, but I would like to know if there is a method to write this data in a streaming manner. I do not need an high compression rate so I could change the algorithm too.

does anyone have an idea about it?

Answer 1

There's a library called TrueZip that I've used with good success in the past to do this kind of thing.

I cannot guarantee it does better on the buffering front. I do know that it does a lot of stuff with its own coding rather than depending on the JDK's Zip API.

So it's worth a try, in my opinion.

Answer 2

ZipOutputStream is stream-based, it doesn't hold onto memory. Your BUFFER may be too large.

Answer 3

I wonder if it's because you are storing the content in a ZipEntry, perhaps it basically loads all of its content before writing out the ZipEntry. Do you have to use Zip? If it's just one data stream you need to compress you might look into the GZIPOutputStream instead. I believe that it would not have the same problem.

Hope this helps.

Answer 4

According to your comment to Sam's response, you have obviously created a ZipOutputStream, which wraps a ByteArrayOutputStream. The ByteArrayOutputStream of course caches the compressed result in memory. If you want it written to disk, you have to wrap the ZipOutputStream around a FileOutputStream.