Background process in GAE

Question

I am developing a website using Google App engine and Django 1.0 (app-engine-patch)

a major part of my program has to run in the background and change local data and also post to a remote url...

can sumone suggest an effective way of doin this...

Answer 1

Without using a third-party system, I think currently your only option is to use the cron functionality.

You'd still be bound by the usual GAE script-execution-time limitations, but it wouldn't happen on a page load.

There is plans for background processing, see this App Engine issue #6, and this roadmap update

Answer 2

I second dbr's recommendation of http://code.google.com/appengine/docs/python/config/cron.html (and hopes for better future approaches, such as the promised "task queues").

Nevertheless I suspect that if you do indeed need major (as in CPU heavy) background processing, GAE may not be the most hospitable environment for that. You may want to consider running those heavy background tasks in other environments, and have them communicate with GAE proper e.g. via the "bulk load/download" APIs, see http://code.google.com/appengine/docs/python/tools/uploadingdata.html (and http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_Data_from_App_Engine for the downloading part).

Google's documentation only describes the usage of the command-line appcfg.py for these purposes (I can't find a proper documentation of the APIs it uses!), but, if you do need more programmatic usage of these APIs, it's not hard to evince them from appcfg.py's sources.

Answer 3

One option is the use the Remote API to access the datastore from another machine (either your own, or something like EC2) to do the processing. The downside to this is that you'll incur bandwidth costs for transferring the data, but you won't be limited to doing your operations in a bunch of 30-second chunks.

Answer 4

Check out The Task Queue Python API that was released a few days ago.