Is it possible to gzip and upload this string to Amazon S3 without ever being written to disk?

Question

I know this is probably possible using Streams, but I wasn't sure the correct syntax.

I would like to pass a string to the Save method and have it gzip the string and upload it to Amazon S3 without ever being written to disk. The current method inefficiently reads/writes to disk in between.

The S3 PutObjectRequest has a constructor with InputStream input as an option.

import java.io.*;
import java.util.zip.GZIPOutputStream;

import com.amazonaws.auth.PropertiesCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.PutObjectRequest;

public class FileStore {

    public static void Save(String data) throws IOException
    {
        File file = File.createTempFile("filemaster-", ".htm");
        file.deleteOnExit();

        Writer writer = new OutputStreamWriter(new FileOutputStream(file));
        writer.write(data);
        writer.flush();
        writer.close();

        String zippedFilename = gzipFile(file.getAbsolutePath());
        File zippedFile = new File(zippedFilename);
        zippedFile.deleteOnExit();

        AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
                new FileInputStream("AwsCredentials.properties")));

        String bucketName = "mybucket";
        String key = "test/" + zippedFile.getName();

        s3.putObject(new PutObjectRequest(bucketName, key, zippedFile));

    }

    public static String gzipFile(String filename) throws IOException
    {
        try {
            // Create the GZIP output stream
            String outFilename = filename + ".gz";
            GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(outFilename));

            // Open the input file
            FileInputStream in = new FileInputStream(filename);

            // Transfer bytes from the input file to the GZIP output stream
            byte[] buf = new byte[1024];
            int len;
            while ((len = in.read(buf)) > 0) {
                out.write(buf, 0, len);
            }
            in.close();

            // Complete the GZIP file
            out.finish();
            out.close();

            return outFilename;
        } catch (IOException e) {
            throw e;
        }
    }


}

Answer 1

I would use something like the following:

ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
GZipOuputStream gzipOut = new GZipOutputStream(byteOut);
// write your stuff
byte[] bites = byteOut.toByteArray();
//write the bites to the amazon stream

You are writing the zipped values out to the byte stream, then taking the byte values, you can write those to your other stream. You can also wrap the stream to the amazon site (i.e. the output stream from the http connection or something similar) and avoid the whole ByteArrayOutputStream.

---

Edit: I noticed your last sentence - bleah. You can take the bytes you created, create a ByteArrayInputStream with them, and then pass that in as an input stream:

ByteArrayInputStream byteInStream = new ByteArrayInputStream(bites);

It should read from the input stream to the output stream, if I am understanding what you are describing correctly. Otherwise, you can simply write to the output stream.

Answer 2

This is essentially what aperkins suggested. I don't know the interface to AS3, so his suggestion to create a ByteArrayInputStream over the byte array is probably the way to go.

import java.io.*; import java.util.zip.GZIPOutputStream;

import com.amazonaws.auth.PropertiesCredentials; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3Client; import com.amazonaws.services.s3.model.PutObjectRequest;

public class FileStore {

public static void Save(String data) throws IOException
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();

Writer writer = new OutputStreamWriter(baos);
    writer.write(data);
    writer.flush();
    writer.close();

byte[] zippedBytes = gzipFile(baos.toByteArray());

   AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
                   new FileInputStream("AwsCredentials.properties")));

   String bucketName = "mybucket";
   String key = "test/" + zippedFile.getName();

   s3.putObject(new PutObjectRequest(bucketName, key, new ByteArrayInputStream(zippedBytes));

}

public static byte[] gzipFile(byte[] bytes) throws IOException
{
    try {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        GZIPOutputStream out = new GZIPOutputStream(baos);

    out.write(bytes, 0, bytes.length);

        // Complete the GZIP file
        out.finish();
        out.close();

    return baos.toByteArray();
    } catch (IOException e) {
        throw e;
    }
}

}