How to stay under GAE quotas? Algorithm design

Question

Hi there

I have a function in my app that uses a lot of resources, and takes time to execute. This is normal and control, however I often get errors due to GAE limit of 30 secs/request.

My function takes the argument and returns several results one after the other, decreasing the size of the argument (a unicode string)

Summary:

def my_function(arg):
    while arg!=u''
        #do_stuff
        #get result
        #arg=new_argument(arg,result)

As the process costs resources, I thought I could split it and enqueue it:

def my_function(arg):
    if arg==u''
       #stop
    else:
      do_stuff
      get_result
      enqueue(my_function(new_argument))

However I'm worried to fall under the API queue limit of 100k calls as I might have a lot of iterations.

I was thinking of redirecting request one to another, which would executes them in a row, but then I have no way to control resource usage:

def my_function(arg):
    if arg==u''
       #stop
    else:
      do_stuff
      get_result
      return redirect('/my_function_url',args=(new_argument))

I don't know if there's a beter way to do it?

Answer 1

I would suggest that you use the Task Queue API, which is perfectly suited to this kind of problems.

Be aware that if you enable billing on your application, you automatically get much larger free quotas : the Task Queue API calls daily limit increases to 20,000,000.

You can set your max daily budget as low as $1 but you won't likely have to pay anything.