Google App Engine GQL query question

Question

in theory, I want to have 100,000 entities in my User model:

I'd like to implement an ID property which increments with each new entity being put.

The goal is so that I can do queries like "get user with IDs from 200 to 300". Kind of like seperating the 10000 entities into 1000 readable pages of 100 entities each.

I heard that the ID property from App Engine does not guarantee that it goes up incrementally.

So how do I implement my own incrementing ID?

One of my ideas is to use memcache. Add a counter in memcache that increases each time a new entity is inserted.

class User(db.Model):
  nickname = db.StringProperty()
  my_id = db.IntegerProperty()


# Steps to add a new user entity:
# Step 1: check memcache
counter = memcache.get("global_counter")

# Step 2: increment counter
counter = counter + 1

# Step 3: add user entity
User(nickname="tommy",my_id=counter).put()

# Step 4: replace incremented counter
memcache.replace(key="global_counter",value=counter)

# todo: create a cron job which periodically takes the memcached global_counter and
# store it into the datastore as a backup ( in case memcache gets flushed )

what do you guys think?

additional question: if 10 users register at the same time, will it mess up the memcache counter?

Answer 1

The memcache counter can go away.

Counters are difficult to do in a distributed system. It is often easiest to re-think the application so that it can do without.

How about using a timestamp instead? You can sort by (and page) on that as well, it will generate the same ordering as a counter.

Answer 2

You don't need to implement your own auto-incrementing counter to achieve the pagination you're looking for - look at "Paging without a property"

In short, the key is guaranteed to be returned in a deterministic order, and you can use inequality operators (>=, <=) to return results starting from a particular key.

To get your first hundred users:

users = User.all().order("__key__").fetch(101)

The first 100 results are what you iterate over; the 101st result, if it is returned, is used as a bookmark in the next query - just add a .filter('__key__ >=', bookmark) to the above query, and you'll get results 200-301