We're thinking of building some of our infrastructure on to Google AppEngine. But we're worried that if it does not scale, we'll need to export the data and run it on our own servers in future.
Is there a way to export Bigtable to MySQL?
+4
A:
As far as data export goes, the Bulk Downloader exists for just this purpose. By default it exports to CSV files, but you can write a custom Exporter class that exports directly to a MySQL database, or any other format of your choosing.
It's also rapidly becoming possible to host an alternate App Engine environment thanks to projects like AppScale, (my own) BDBDatastore, and TwistedAE.
In general, I think your scaling concerns are unfounded - App Engine already hosts many apps that receive high levels of traffic, but that depends in part on your app and its needs. Of course, I'm biased - I'm on the App Engine team, so take my assessment with a grain of salt. ;)
Nick Johnson
2009-08-23 12:13:39
I'd note that GAE scales, *if* you can write your program to use the datastore/BigTable "properly". But if you're doing work in Python to filter or compare result sets, that SQL is capable of doing in the database but BigTable is not, or if you use wide-ranging transactions, then your app quite possibly will scale better on a single-machine LAMP stack than it will on GAE.
Steve Jessop
2009-08-23 13:15:43
Hence my comment "but that depends in part on your app and its needs". As far as filtering stuff in Python goes, although it's less efficient, it's only by a constant factor - doing it in the DB is still the same big-O runtime, so if it doesn't scale filtering in Python, it won't scale filtering linearly in the DB.
Nick Johnson
2009-08-23 20:18:06