Strategy for interleaving jobs on AppEngine?

Question

Let's say I have 1000's of jobs to perform repeatedly, how would you propose I architect my system on Google AppEngine?

I need to be able to add more jobs whilst effectively scaling the system. Scheduled Tasks are part of the solution of course as well as Task Queues but I am looking for more insights has to best utilize these resources.

NOTE: There are no dependencies between "jobs".

Answer 1

Hi,

I think you could use Cron Jobs.

Regards.

Answer 2

Based on what little description you've provided, it's hard to say. You probably want to use the Task Queue, and maybe the deferred library if you're using Python. All that's required to use these is to use the API to enqueue a task.

If you're talking about having many repeating tasks, you have a couple of options:

• Start off the first task on the task queue manually, and use 'chaining' to have each invocation queue the next one with the appropriate interval.
• Store each schedule in the datastore. Have a cron job regularly scan for any tasks that have reached their ETA; fire off a task queue task for each, updating the ETA for the next run.