Google App Engine Put() Timeout

Question

Hi i've searched the App Engine Groups but couldn't find a definitive solution.

I want to put 1000 entities into my App Engine Server (Production Server), but I get timeout errors and also 500 Server errors. It worked perfectly in local development.

To reduce the Put load, I made the code sleep for 5 seconds after every 10 puts. I still got the same errors :(

Please point me in the right direction. Really appreciate your insights.

Code:

class User(db.Model)
   nickname = db.StringProperty()
   feed = db.StringListProperty()

class Initialize(webapp.RequestHandler):
  def get(self):
    self.response.headers['Content-Type'] = 'text/plain'
    start = 0
     end = 10

  #add 1000 users ( 10 Puts x 100 rows )
  for row in range(1,100):
    for i in range(start,end):
      nickname = "nickname" + str(i)
      feed = ["string1","string2","string3","string4","string5"]

      User(key_name=str(i),nickname=nickname,feed=feed).put()

      self.response.out.write('Entry %s done\n' % i)

    #add counters by 10
    start = start + 10
    end = end + 10

    #sleep
    time.sleep(5)

  self.response.out.write('Initialized 1000 users for Datastore.')

Answer 1

I made the code sleep for 5 seconds after every 10 puts

That will not work, because your request can only take a certain amount of time before it time-outs. Going to sleep will just count against that limit.

You need to split your operation across multiple requests, for example using a task queue.

Answer 2

Something else to consider is that GAE sometimes errors/times out on the first request after a period of app quiescence. Seems it takes a while to "wake up" the app if it's been inactive for some hours, so the first request after such a time errors almost automatically.

Have you tried re-doing your request after the initial failure?

Answer 3

I would suggest that you need to break up the writes into batches as write operations use a lot of time, and you will otherwise exceed AppEngine's maximum CPU usase per request. The correct way to break into into batches is to have multiple, separate requests that each add in a small number of records.

So code the Initialize handler such that you can call it multiple times, each call accomplishing a small piece of the total work.

Answer 4

Datastore timeouts are more likely for large puts, so the easiest solution is to store records in smaller batches. If you're also running out of execution time (a request timeout), you need to use the task queue to break your operation into many small requests.