Unzipping a BlobstoreInputStream >= 1MB in size

Question

I've got a small function that reads a zip file from an input stream and then stores the contents in an arraylist. I'm using GAE which is limited to files less than 1mb. I've converted the InputStream into a BlobstoreInputStream so I can now upload zip files that are > 1mb. However I can still only access < 1Mb of data from it or I get an error*. I think I need to chain together multiple <1mb reads of the BlogstoreInputStream but I don't know how to do this.

My code which works for any zip file < 1mb in size is below. No file within the zip file will ever be more than 1mb.

I'm not too clued up on inputstreams so if anyone can help me with this or point me in the right direction I will appreciate it.

private void setZipItems() throws IOException, Exception {
    BlobstoreInputStream in = new BlobstoreInputStream(blobKey);
    ZipInputStream zis = new ZipInputStream(new BufferedInputStream(in));
    ZipEntry entry;
    while ((entry = zis.getNextEntry()) != null) {
         ZipItem item = new ZipItem();
         item.fileName = entry.getName();
         ByteArrayOutputStream contents = new ByteArrayOutputStream();
         int size;
         byte[] buffer = new byte[1024];
         while ((size = zis.read(buffer)) != -1) {
             if (size != buffer.length) { 
                 for (int i = 0 ; i < size; i ++)
                     contents.write(buffer[i]);
             } else contents.write(buffer);
          }
          zis.closeEntry();
          item.contents = contents.toByteArray();
          zipItems.add(item);
    }
    zis.close();
    in.close();
}

*BlobstoreInputStream BlobstoreIOException: "Blob Fetch Size too large" and occurs at

while ((entry = zis.getNextEntry()) != null).

On first run with blobstore entities larger than 1mb.

Answer 1

while ((size = zis.read(buffer)) > 0)
  contents.write(buffer, 0, size);

But the whole concept of reading entire files into memory has knobs on it. Find a way to process it as you read it.

or I get an error.

What error?

Answer 2

Ok,I solved it a couple of days ago. The trick is not to try to unzip the BlobstoreInputStream which can only read 1mb of the file at a time but rather to read the entire file into a byte array using BlobstoreService.fetchData() and then to unzip the byte array.

In case anyone else has trouble with this, I've included my entire code below. It is truly horrible code that I'm not very proud of but it gets the job done.

private void setZipItems() throws IOException, Exception {
    final int readSize = 1015800;  //max blobstore readsize
    byte[] blob = null;
    BlobInfo bi = new BlobInfoFactory().loadBlobInfo(blobKey);
    long count = 0;
    BlobstoreService bs = BlobstoreServiceFactory.getBlobstoreService();
    while ((bi.getSize() - count) > 0) {

        if ((count + readSize) < bi.getSize()) {
            if (blob == null) blob = bs.fetchData(blobKey, count, count + readSize - 1);
            else blob = concat(blob, bs.fetchData(blobKey, count, count + readSize - 1));
            count = count + readSize;
        } else {
            if (blob == null) blob = bs.fetchData(blobKey, count, bi.getSize() - 1);
            else blob = concat(blob, bs.fetchData(blobKey, count, bi.getSize() - 1));
            break;
        }

     }
    bs.delete(blobKey);

    //The above code writes the entire Blobstore entity to a byte Array.  Below I then transform it into an inputstream so that I can plug it into my original function. This is horribly inefficient since byte arrays can be unzipped directly but I'm pretty lazy so don't mock me to much.
            ByteArrayInputStream in = new ByteArrayInputStream(blob);
    ZipInputStream zis = new ZipInputStream(new BufferedInputStream(in));
    ZipEntry entry;

    while ((entry = zis.getNextEntry()) != null) {
        if (unrecognizedFile(entry.getName())) continue;
        ZipItem item = new ZipItem();
        item.fileName = entry.getName();

        ByteArrayOutputStream contents = new ByteArrayOutputStream();
        int size;
        byte[] buffer = new byte[1024];
        while ((size = zis.read(buffer)) != -1) {
            if (size != buffer.length) { 
                for (int i = 0 ; i < size; i ++)
                    contents.write(buffer[i]);
            } else  contents.write(buffer);
        }
        zis.closeEntry();
        item.contents = contents.toByteArray();

        zipItems.add(item);

    }

    zis.close();
    in.close();
 }