S3: Duplicate buckets

Question

What is the easiest way to duplicate an entire Amazon S3 bucket to a bucket in a different account? Ideally, we'd like to duplicate the bucket nightly to a different account in Amazon's European data center for backup purposes.

Answer 1

I suspect there is no "automatic" way to do this. You'll just have to write a simple app that moves the files over. Depending on how you track the files in S3 you could move just the "changes" as well.

On a related note, I'm pretty sure Amazon does a darn good job backup up the data so I don't think you necessarily need to worry about data loss, unless your back up for archival purposes or something.

Answer 2

One thing to consider is that you might want to have whatever is doing this running in an Amazon EC2 VM. If you have your backup running outside of Amazon's cloud then you pay for the data transfer both ways. If you run in an EC2 VM, you pay no bandwidth fees (although I'm not sure if this is true when going between the North American and European stores) - only for the wall time that the EC2 instance is running (and whatever it costs to store the EC2 VM, which should be minimal I think).

Answer 3

Cool, I may look into writing a script to host on Ec2. The main purpose of the backup is to guard against human error on our side -- if a user accidentally deletes a bucket or something like that.

Answer 4

If you're worried about deletion, you should probably look at S3's new Versioning feature.