Upload file or InputStream to S3 with a progress callback

Question

We are uploading a file using the Amazon AWS Java Library and are having difficulty obtaining upload progress. We're currently calling the following:

File file = new File(localAsset.getVideoFilePath());
PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, localAsset.getFileName(), file);
s3.putObject(putObjectRequest);

How can we set a callback to check up on file upload progress?

Thanks

Answer 1

There's currently no easy way to do that "built-in", but you can easily implement one by wrapping the InputStream you pass in to report on how far it's being read.

You can read the team's official response to this question on their forums.

It might also be worth noting that the .NET SDK does have this feature, though as far as I know it's implemented in a similarly "hackish" way (simply, the S3 API itself doesn't really have built-in provisions for doing this gracefully). If you're having trouble implementing it yourself, that might be a place to look to for inspiration.

Answer 2

I came upon this exact problem and wrote a simple InputStream wrapper that prints out nice progress bars:

import java.io.IOException;
import java.io.InputStream;

import org.apache.commons.vfs.FileContent;
import org.apache.commons.vfs.FileSystemException;

public class ProgressInputStream extends InputStream {
    private final long size;
    private long progress, lastUpdate = 0;
    private final InputStream inputStream;
    private final String name;
    private boolean closed = false;

    public ProgressInputStream(String name, InputStream inputStream, long size) {
        this.size = size;
        this.inputStream = inputStream;
        this.name = name;
    }

    public ProgressInputStream(String name, FileContent content)
    throws FileSystemException {
        this.size = content.getSize();
        this.name = name;
        this.inputStream = content.getInputStream();
    }

    @Override
    public void close() throws IOException {
        super.close();
        if (closed) throw new IOException("already closed");
        closed = true;
    }

    @Override
    public int read() throws IOException {
        int count = inputStream.read();
        if (count > 0)
            progress += count;
        lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size);
        return count;
    }
    @Override
    public int read(byte[] b, int off, int len) throws IOException {
        int count = inputStream.read(b, off, len);
        if (count > 0)
            progress += count;
        lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size);
        return count;
    }

    static long maybeUpdateDisplay(String name, long progress, long lastUpdate, long size) {
        if (Config.isInUnitTests()) return lastUpdate;
        if (size < B_IN_MB/10) return lastUpdate;
        if (progress - lastUpdate > 1024 * 10) {
            lastUpdate = progress;
            int hashes = (int) (((double)progress / (double)size) * 40);
            if (hashes > 40) hashes = 40;
            String bar = StringUtils.repeat("#",
                    hashes);
            bar = StringUtils.rightPad(bar, 40);
            System.out.format("%s [%s] %.2fMB/%.2fMB\r",
                    name, bar, progress / B_IN_MB, size / B_IN_MB);
            System.out.flush();
        }
        return lastUpdate;
    }
}

(this is copy-and-pasted from live code, so you might have to do a few fixups to get it to work in your own code.)

Then, just use the InputStream way of putting things (make sure to specify the size!) and it will make a nice progress bar for you. If you want a proper callback that'd be pretty easy to do too.