Db Backup Options (for Cloud DBs like Amazon ESB and GAE)

views:

213

answers:

Db Backup Options (for Cloud DBs like Amazon ESB and GAE)

What are db backup options for (offline):

1 - Amazon EC2

2 - GAE (Google App Engine)

Note: For example ESB (Amazon Elastic Block Store) provides differential snapshots which is great; and we can have backups in Amazon S3, which is great too. But I do not know if I can have latest version of my db, on my local machine.

Thanks

+2 A:

For App Engine, the bulkloader permits dumping and loading the entire datastore. Loading to a different instance that you dumped from is also possible, but currently requires manually setting the ID counters using allocate_ids so that they don't allocate already used IDs. Future versions of the SDK should support doing this automatically.

Asking about backing up EC2 doesn't make much sense - EC2 is for computation, not storage. If you mean one of the other services - S3, EBS, or SimpleDB, you need to specify which you mean.

Nick Johnson 2009-10-25 13:34:09

Thanks for your answer.1 - Google: According to http://code.google.com/appengine/articles/gae_backup_and_restore.html (Which is an effort for implementing a Backup and Restore solution for GAE) "Google App Engine does not come with a data backup solution" but "Google is currently working on its own datastore import and export utility for large datasets which is currently on the roadmap for the first quarter of 2009"; this is what I have found about it.

Kaveh Shahbazian 2009-10-25 20:12:54

2 - Amazon: I am aware that EC2 is just for computation (Running your web/application server or database engine). I have put it at the title because I want to know if anybody had good experiences with using local databases on EC2 itself.Yet in Amazon family, I was asking mainly about ESB; because I want to have an Oracle 11g with a - currently - 40 GB of data - which is very important for me to have daily backups and have weekly "hard" backups.

Kaveh Shahbazian 2009-10-25 20:14:42

1) The article is out of date - as you can see from the link I provided, App Engine now has dump/restore support.If you want to back up an EBS volume on Amazon, you can take a snapshot of it, or you can download it - back it up in the same manner you would any other filesystem on any other system.

Nick Johnson 2009-10-25 22:25:14

Thanks Nick for the link; when I saw it at first a question came up: how can I backup all of my database?At a note at the same page Google said: "This feature is experimental and currently in the testing stages". So I take it as "Google will have a backup/restore" solution; yet not now.

Kaveh Shahbazian 2009-10-26 07:16:02

Experimental, but it works right now.

Nick Johnson 2009-10-26 13:30:38

Thanks Nick; it seems you have good experiences with GAE. Lack of aggregation operations and joins are a real pain. But I ask a question: Will you put a 50 GB mission-critical data on GAE? Have you had a confident feeling about GAE; which amounts to an important project for you? As a side note, most part of these data are daily updating GPS information.

Kaveh Shahbazian 2009-10-26 13:48:17

I would happily do so, but I'm biased - I'm on the App Engine team. :)

Nick Johnson 2009-10-27 21:36:15

+1 A:

You're quite right to ask about EC2 backup, as running SQL Server (for example) on there is a typical use case. It's normal practice to use Amazon's Elastic Block Store (EBS) as the storage for the data files in that case, and EBS can easily be backed to by taking snapshots. However, if you have large and busy database, note that you will not be able to take a successful snapshot of the live data drive in a sensible amount of time. Instead, it's necessary to use a second EBS volume as a backup volume and do conventional database backups from the live volume to the backup volume. The snapshot of the backup volume can then run uncontended after the database backup is complete.

Nestor 2009-10-26 09:05:32

That's what I'v thought :( Then it will not help me so much with eliminating some of server housekeeping. And SimpleDB - even if I decide to redesign everything - has limitations; i.e. my message table alone is over 25 GB and it's all GPS data! But max size for a domain in SimpleDB is 10 GB! And I will have other kind of problems with limitations of GAE database limitations! Maybe I should give up on cloud and return to my old fashioned web application. Thanks

Kaveh Shahbazian 2009-10-26 13:17:02

My word, have a look at this (new out today): http://aws.typepad.com/aws/2009/10/introducing-rds-the-amazon-relational-database-service-.html

Nestor 2009-10-27 07:13:17

Thanks maxc; now that's more interesting

Kaveh Shahbazian 2009-10-27 11:06:58

You can definitely set up EC2 to have easy and convenient snapshots of EBS volumes. The EBS volume can be up to 1TB. Check out this article for how to set up MySQL to use EBS: ec2ebs-mysql.notlong.com/ And this article for a script (and explanation) to create consistent snapshots of the EBS volume: alestic.com/2009/09/ec2-consistent-snapshot

Shlomo Swidler 2009-11-08 11:03:30

ansaurus

tags:

views:

answers:

Db Backup Options (for Cloud DBs like Amazon ESB and GAE)

related questions