GAE Query counter (+1000)

Question

How to implement this query counter into existing class? The main purpose is i have datastore model of members with more than 3000 records. I just want to count its total record, and found this on app engine cookbook:

def query_counter (q, cursor=None, limit=500):
    if cursor:
        q.with_cursor (cursor)
    count = q.count (limit=limit)
    if count == limit:
        return count + query_counter (q, q.cursor (), limit=limit)
    return count

My existing model is:

class Members(search.SearchableModel):
    group = db.ListProperty(db.Key,default=[])
    email = db.EmailProperty()
    name = db.TextProperty()
    gender = db.StringProperty()

Further i want to count members that join certain group with list reference. It might content more than 1000 records also.

Anyone have experience with query.cursor for this purpose?

Answer 1

To find all members you would use it like this:

num_members = query_counter(Members.all())

However, you may find that this runs slowly, because it's making a lot of datastore calls.

A faster way would be to have a separate model class (eg MembersCount), and maintain the count in there (ie add 1 when you create a member, subtract 1 when you delete a member).

If you are creating and deleting members frequently you may need to create a sharded counter in order to get good performance - see here for details:

http://code.google.com/appengine/articles/sharding_counters.html

To count members in a certain group, you could do something like this:

group = ...
num_members = query_counter(Members.all().filter('group =', group.key()))

If you expect to have large numbers of members in a group, you could also do that more efficiently by using a counter model which is sharded by the group.