<class 'google.appengine.runtime.DeadlineExceededError'>: how to get around?

Question

Ok guys I am having tons of problems getting my working dev server to a working production server :). I have a task that will go through and request urls and collect and update data. It takes 30 minutes to run.

I uploaded to production server and going to the url with its corresponding .py script appname.appspot.com/tasks/rrs after 30 seconds I am getting the class google.appengine.runtime.DeadlineExceededError' Is there any way to get around this? Is this a 30 second deadline for a page? This script works fine in development server I go to the url and the associate .py script runs until completion.

import time
import random
import string
import cPickle
from StringIO import StringIO
try:
    import json
except ImportError:
    import simplejson as json 
import urllib
import pprint
import datetime
import sys
sys.path.append("C:\Program Files (x86)\Google\google_appengine")
sys.path.append("C:\Program Files (x86)\Google\google_appengine\lib\yaml\lib")
sys.path.append("C:\Program Files (x86)\Google\google_appengine\lib\webob")
from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import db
class SR(db.Model):
    name = db.StringProperty()
    title = db.StringProperty()
    url = db.StringProperty()

##request url and returns JSON_data
def overview(page):
     u = urllib.urlopen(page)
     bytes = StringIO(u.read())
     ##print bytes
     u.close()
     try:
        JSON_data = json.load(bytes)
        return JSON_data
     except ValueError,e:
        print e," Couldn't get .json for %s" % page
        return None

##specific code to parse particular JSON data and append new SR objects to the given url list
def parse_json(JSON_data,lists):
    sr = SR()
    sr.name = ##data gathered
    sr.title = ##data gathered
    sr.url = ##data gathered
    lists.append(sr)
    return lists

## I want to be able to request lets say 500 pages without timeing out
page = 'someurlpage.com'##starting url
url_list = []
for z in range(0,500):
    page = 'someurlpage.com/%s'%z
    JSON_data = overview(page)##get json data for a given url page
    url_list = parse_json(JSON_data,url_list)##parse the json data and append class objects to a given list
db.put(url_list)##finally add object to gae database

Answer 1

Yes, the App Engine imposes a 30 seconds deadline. One way around it might be a try/except DeadlineExceededError and putting the rest in a taskqueue.

But you can't make your requests run for a longer period.

You can also try Bulkupdate

Example:

class Todo(db.Model):
    page = db.StringProperty()

class BulkPageParser(bulkupdate.BulkUpdater):
    def get_query(self):
        return Todo.all()

    def handle_entity(self, entity):
        JSON_data = overview(entity.page)
        db.put(parse_json(JSON_data, [])
        entity.delete()

# Put this in your view code:
for i in range(500):
    Todo(page='someurlpage.com/%s' % i).put()

job = BulkPageParser()
job.start()

Answer 2

ok so if I am dynamically adding links as I am parsing the pages, I would add to the todo queue like so I believe.

def handle_entity(self, entity):
    JSON_data = overview(entity.page)
    data_gathered,new_links = parse_json(JSON_data, [])##like earlier returns the a list of sr objects, and now a list of new links/pages to go to
    db.put(data_gathered)
    for link in new_links:
        Todo(page=link).put()
    entity.delete()