Use EC2 to Zip S3 files

Question

Hello all,

I am trying to use EC2 to zip up some files that are stored in an S3 bucket. I have gotten as far as successfully getting SWFUpload to work with PHP and upload the files to S3. I read that the best way to zip up S3 files without incurring huge transfer costs is to use EC2 to deal with S3. After a lot of effort I managed to get a EC2 server running and SSH into it, but now I don't know what to do from here.

What's the best way to zip S3 files and put them back in the bucket?

Ideally, a user's batch upload would trigger an SQS and then I'd spin up the EC2 server once a day to zip them all and hand them back to S3 for downloading. No idea where to go from here. Ideas?

Answer 1

You should check out aws - a command line tool to ec2, s3 and other AWS services.

When you install the tool and setup your AWS credentials, it will create handy symlinks all kinds of commands -- including s3.

The general idea is:

• download the file from the bucket (s3get <bucket/file>)
• zip it (gzip <file>)
• upload it again (s3put <file>)

Let me know if you need more pointers.

Maybe you want to zip the file in PHP before you upload them? This requires the zip extension to be installed.

sudo pecl install zip

Here is an example script:

<?php
$zip      = new ZipArchive();
$filename = "/tmp/" . time() .".zip";
if ($zip->open($filename, ZIPARCHIVE::CREATE) !== true) {
    exit("cannot open <$filename>\n");
}
$zip->addFile('/path/to/uploaded/file');
$zip->close();

// continue uploading to s3