Retrieving hierarchial data from datastore

Question

Hi!

I am building an app with Google App Engine (and Python). I have two questions about retrieving data from datastore.

I have information about users and information about posts made by users. Here is part of DB Models:

class User(db.Model):
    country = db.StringProperty() 
    # many other entities

class Post(db.Model):
    author = db.ReferenceProperty(User) 
    # many other entities

I want to retrieve Posts, that are made by users from certain country. I tried it this way:

posts_query = Post.all().filter(' author.country == ', country)
posts = posts_query.fetch(limit = 100)

But this query does not return any results (the country variable has a value that exists in the datastore). What I need to correct in my query, so it works?

And the second question: how (in the given situation) I can retrieve all posts from datastore, if the post count is unknown (and may be > 100)?

Thanks in advance,

-skazhy

Answer 1

If you want to do this, you need to denormalize: store a copy of the user's country on all their posts, and query on that. The datastore doesn't support joins, which would be required to satisfy this query, and in any case denormalizing is a lot more efficient.

As far as retrieving 'all' results goes, you can specify an arbitrarily large limit, but consider: do you really want to do this? There's a limit on how many results you can sensibly present to a user in one go; if you have more results than that, you almost certainly want to paginate, instead.

Answer 2

Seconding denormalization. If country is a specific thing you need to query on frequently, you could add a new model just for it. The Country then becomes a ReferenceProperty for both the user and the Post:

class User(db.Model):
  country = db.ReferenceProperty(Country,collection_name=users) 
  # many other entities

class Post(db.Model):
  author = db.ReferenceProperty(User) 
  country = db.ReferenceProperty(Country,collection_name=posts) 
  # many other entities

class Country(db.Model):
  name = db.StringProperty()

# posts from everyone in the user's country:
# user.country.posts.all()

In response to your second question, you can use your posts_query as an iterable to retrieve all results. See the App Engine docs: http://code.google.com/appengine/docs/python/datastore/queryclass.html#Query_fetch