timeout workarounds in app engine

Question

I am building a demo for a banking application in App Engine. I have a Users table and Stocks table.

In order for me to be able to list the "Top Earners" in the application, I save a "Total Amount" field in each User's entry so I will later be able to SELECT it with ORDER BY.

I am running a cron job that runs over the Stocks table and update each user's "Total Amount" in the User's table. The problem is that I often get TIMEOUTS since the Stocks table is pretty big.

Is there anyway to overcome the time limit restriction in App Engine, or is there any workaround for these kind of updates (where you MUST select many entries from a table that result a timeout)?

Joel

Answer 1

The usual way is to split the job into smaller tasks using the task queue.

Answer 2

You have several options, all will involve some form of background processing.

One choice would be to use your cron job to kick off a task which starts as many tasks as needed to summarize your data. Another choice would be to use one of Brett Slatkin's patterns and keep the data updated in (nearly) realtime. Check out his high performance data-pipelines talk for details.

http://code.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html

You could also check out the mapper api (app engine map reduce) and see if it can do what you need.

http://code.google.com/p/appengine-mapreduce/