Breaking out of the Google App Engine Python lock-in?

Question

Are there any guidelines to writing Google App Engine Python code that would work without Google's infrastructure on other platforms?

Is there any known attempt to create an open source framework which can run applications designed for Google App Engine on other platforms?

Edit:

To clarify, the question really is:

If I develop an application on Google App Engine now, will I be able to migrate to another platform later, or is it a lock in?

Answer 1

The code should be mostly portable (they do a pretty good job of indicating which modules you can't use on AppEngine and which AppEngine-specific code corresponds to which forbidden modules), but the whole point of AppEngine is to get access to Google's infrastructure. There's not much point to writing your code to the AppEngine restrictions if you aren't going to be using their infrastructure.

Answer 2

So far, I found an experimental host called app-drop which is capable of hosting google app-engine projects. This should mean that it's atleast possible to run app engine projects outside of google's infrastructure.

This is however clearly not yet suitable for production.

Answer 3

You can build AppEngine applications using the Django python framework (although the supported version is a bit behind the most recent Django release). Where you lose portability (at least right now) is when using GQL/BigTable for persistence. This is Google proprietary database platform. As Hank mentioned this is one of the biggest reasons to actually use AppEngine, but it is also the single largest locking point.

Here are a couple of links to Django support in AppEngine and GQL/BigTable:

• http://code.google.com/appengine/articles/django.html
• http://code.google.com/appengine/docs/python/datastore/gqlreference.html

Answer 4

Use a high level framework that works on App-Engine. That way you can port your code to other servers when you want.

django has been patched and ported to work in the Appengine patch project and is the most used FW on appengine.

You may want to refer this step by step intro to running a django app on App engine

As far as the parallel infrastructure to run an app engine application is concerned, it is still way far. App Engine itself hasn't got as popular as people believed it to and google wanted it to be. Plus it is harder to develop on the builtin WebApp framework than on django.

Its quite unlikely to see a parallel infrastructure to run app engine application on, atleast for years to come. Rather it is likely to see django and other popular frameworks work out of the box on app engine and the work on this is currently underway in the referred project.

Answer 5

AppDrop is a proof of concept port of AppEngine to Amazon Web Services / Elastic Computing completed in April of 2008. It uses a flat file instead of BigTable and runs in a single instance, so there are scaling issues; but it's developer says it took him only four days, and perhaps these limitations can be addressed by others.

Answer 6

I'm dealing with the same problem, and the only real problem is the datastore. I mean, the memcache and wsgi app have the exact same interfaces as outside appengine.

So I abstracted the memcache in some cache module where I can replace the backend (pretty much like django does it).

And I'm planning on implementing a subset of the datastore api objects and methods (basically get, put, query, Model) on top of a different backend like tokyo cabinet. You could even do this on top of an sql database with a smart data model, but I wouldn't recommend it.

Answer 7

There's a number of components required to make an app fully portable:

• The runtime environment itself. This can be ported relatively simply, by setting up a CGI or FastCGI server that emulates the App Engine environment (which itself is basically slightly-enhanced CGI). Most of the code to do this is already in the SDK. Unfortunately, there's no easy pre-packaged toolkit for this yet.
• The datastore. By far the most complicated API to port. There are a number of efforts underway: AppScale runs on EC2/Eucalyptus/Xen and uses a HyperTable or HBase backend; it's still very much beta quality and they don't distribute the data connector separately (it's the beginnings of a complete run-your-app-on-your-own-cloud solution). Jens is/was writing an SQLite backend, and there's my own project, BDBDatastore, which uses BDB-JE as the backend, and is fully functional (though beta quality). AppDrop, which others have mentioned, simply uses the development server as a backend, and hence isn't suitable for production use.
• The users API needs replacing with something else, such as an OpenID based API. Again, fairly simple, but no premade solutions yet.
• The Memcache API needs a backend that uses standard C memcache backends.
• The other APIs have perfectly functional backends as part of the SDK, so don't really need porting.
• Cron support would also need implementing, as would background processing, XMPP, etc, when they become available.

As you can see, there's a lot of work to be done, but no fundamental barriers to making your App Engine app run outside Google's environment. In fact, if you're interested, you're more than welcome to participate - I and others have plans to combine the solutions for the various pieces into a single 'OpenEngine' solution for hosting your own apps.

Answer 8

I did the reverse migration from vanilla Unix to app engine recently very easily by using WHIFF resources. Basically configure anything platform dependant as a resource and then swap/replace the resources on different configurations.

http://piopio.appspot.com/W1000_1000.resources

also see

http://aaron.oirt.rutgers.edu/myapp/docs/W1100_1200.wwiki

for a detailed example of resource swapping/configuration. (note: links may go away eventually, app is experimental.)

Answer 9

Check out typhoonae. it's in beta, but quite usable – we moved one of our apps to inhouse server running this stack.