google appengine/python: Can I depend on Task queue retry on failure to keep insertions to a minimum?

Question

Say my app has a page on which people can add comments. Say after each comment is added a taskqueue worker is added. So if a 100 comments are added a 100 taskqueue insertions are made.

(note: the above is a hypothetical example to illustrate my question)

Say I wanted to ensure that the number of insertions are kept to a minimum (so I don't run into the 10k insertion limit)

Could I do something as follows.

a) As each comment is added call taskqueue.add(name="stickytask", url="/blah") - Since this is a named taskqueue it will not be re-inserted if a taskqueue of the same name exists.

b) The /blah worker url reads the newly added comments, processes the first one and than if more comments exist to be processed returns a status code other than 200 - This will ensure that the task is retried and at next try will process the next comment and so on.

So all 100 comments are processed with 1 or a few taskqueue insertion. (Note: If there is a lull in activity where no new comments are added and all comments are processed than the next added comment will result in a new taskqueue insertion. )

However from the docs (see snippet below) it notes that "the system will back off gradually". Does this mean that on each "non 200" Http status code returned a delay is inserted into the next retry?

From the docs: If the execution of a particular Task fails (by returning any HTTP status code other than 200 OK), App Engine will attempt to retry until it succeeds. The system will back off gradually so as not to flood your application with too many requests, but it will retry failed tasks at least once a day at minimum.

Answer 1

There's no reason to fake a failure (and incur backoff &c) -- that's a hacky and fragile arrangement. If you fear that simply scheduling a task per new comment might exceed the task queues' currently strict limits, then "batch up" as-yet-unprocessed comments in the store (and possibly also in memcache, I guess, for a potential speedup, but, that's optional) and don't schedule any task at that time.

Rather, keep a cron job executing (say) every minute, which may deal with some comments or schedule an appropriate number of tasks to deal with pending comments -- as you schedule tasks from just one cron job, it's easy to ensure you're never scheduling over 10,000 per day.

Don't let task queues make you forget that cron is also there: a good architecture for "batch-like" processing will generally use both cron jobs and queued tasks to simplify its overall design.

To maximize the amount of useful work accomplished in a single request (from ether a queued task or a cron one), consider an approach based on monitoring your CPU usage -- when CPU is the factor limiting the work you can perform per request, this can help you get as many small schedulable units of work done in a single request as is prudently feasible. I think this approach is more solid than waiting for an OverQuotaError, catching it and rapidly closing up, as that may have other consequences out of your app's control.