Transfer files between Amazon EC2 and S3

Question

I have log files generated by my application running on EC2. I want to first upload the log file to a bucket in S3. I then run a map reduce job and I want to then download the output file back to EC2.

What is the best way for me to upload and download files from EC2 and S3? Is it possible to do automate the process using Shell script on EC2, given that I'm using a linux instance? Also, would I need to modify access permission on the s3 bucket?

Answer 1

You can use the amazon rest tools to accomplish this. You need your access key and security keys. Using those you can use the tools that are usually already installed on the box to post a file to the s3 server.

To fetch it, you can just use wget.

Answer 2

This is another tool I'm using for s3: http://s3tools.org/s3cmd