Datastore performance, my code or the datastore latency

Question

I had for the last month a bit of a problem with a quite basic datastore query. It involves 2 db.Models with one referring to the other with a db.ReferenceProperty.

The problem is that according to the admin logs the request takes about 2-4 seconds to complete. I strip it down to a bare form and a list to display the results. The put works fine, but the get accumulates (in my opinion) way to much cpu time.

#The get look like this: 
outputData['items'] = {} 
labelsData = Label.all() 
for label in labelsData: 
        labelItem = label.item.name 
        if labelItem not in outputData['items']: 
                outputData['items'][labelItem] = { 'item' : labelItem, 'labels' : [] } 
        outputData['items'][labelItem]['labels'].append(label.text) 
path = os.path.join(os.path.dirname(__file__), 'index.html') 
self.response.out.write(template.render(path, outputData)) 
#And the models: 
class Item(db.Model): 
        name = db.StringProperty() 
class Label(db.Model): 
        text = db.StringProperty() 
        lang = db.StringProperty() 
        item = db.ReferenceProperty(Item) 

I've tried to make it a number of different way ie. instead of ReferenceProperty storing all Label keys in the Item Model as a db.ListProperty.

My test data is just 10 rows in Item and 40 in Label.

So my questions: Is it a fools errand to try to optimize this since the high cpu usage is due to the problems with the datastore or have I just screwed up somewhere in the code? ..fredrik

EDIT:

I got a great response from djidjadji at the google appengine mailing list.

The new code looks like this:

outputData['items'] = {}
labelsData = Label.all().fetch(1000)
labelItems = db.get([Label.item.get_value_for_datastore(label) for label in labelsData ])
for label,labelItem in zip(labelsData, labelItems):
   name = labelItem.name
   try:
       outputData['items'][name]['labels'].append(label.text)
   except KeyError:
       outputData['items'][name] = { 'item' : name, 'labels' : [label.text] }

Answer 1

There's certainly things you can do to optimize your code. For example, you're iterating over a query, which is less efficient than fetching the query and iterating over the results.

I'd recommend using Appstats to profile your app, and check out the Patterns of Doom series of posts.

Answer 2

Don't just try things. That's guessing. You'll only be right some of the time. Don't ask other people to guess either, for the same reason.

Be right every time.

Just pause the code several times and look at the call stack. That will tell you exactly what's going on.