views:

58

answers:

2

I have a query that is very recently starting to throw:

"The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query."

I checked the line on which this exception is being thrown, and the problem query is this one:

count = self.vote_set.filter("direction =", 1).count()

This is literally a one-filter operation using appengine's built-in backreferences. I have no idea how to optimize this query...anyone have any suggestions? I tried to add this index:

 - kind: Vote
   properties:
   - name: direction
     direction: desc

 - kind: Vote
   properties:
   - name: direction

And I got a message (obviously) saying this was an unnecessary index.

Thanks for your help in advance.

+3  A: 

The backreferences actually just construct a query that's filtered on the reference property, so by adding another filter you have a 2-filter query.

Your compsite index would look something like:

- kind: Vote
  properties:
  - name: your_reference_property_name
  - name: direction
    direction: desc
Wooble
After waiting two days for this index to build, it seems to have no effect at all. After checking out this page:http://www.allbuttonspressed.com/blog/django/2010/01/An-App-Engine-limitation-you-didn-t-know-aboutI'm thinking I might not be able to stick with this schema. Argh.See, the painful part is, this is a live site that's being used, it's not just a hobby project. It's making money, so now I have to work around this limitation and waste time. I'm considering just porting this to run on a normal server of my own.
dave paola
+3  A: 

If you run all relevant queries on your local SDK, it should generate all needed indices (in index.yaml) and the recommended policy is not to edit index.yaml yourself but rather to let the local SDK do it for you. If the SDK is not generating all needed indices, as long of course as you do exercise all relevant code paths in your local testing!, you should open a bug for it in the App Engine tracker here (after checking that there isn't already a bug report for this problem, of course).

Alex Martelli
See, I've definitely done this, so it makes me think the size of my datastore has something to do with it. I'm currently waiting on some new indices to build, but after that, I may have to open a bug report. Very strange.
dave paola
The SDK won't generate indexes for queries that can be executed with the merge join algorithm. It's possible for a query to be impractical to do with a merge-join, though - which is what we're seeing here.
Nick Johnson
So Nick: is there anything I can do besides redesigning my schema?
dave paola