SelfReferenceProperty vs. ListProperty Google App Engine

Question

Hi All,

I am experimenting with the Google App Engine and have a question.

For the sake of simplicity, let's say my app is modeling a computer network (a fairly large corporate network with 10,000 nodes). I am trying to model my Node class as follows:

class Node(db.Model):
    name = db.StringProperty()
    neighbors = db.SelfReferenceProperty()  

Let's suppose, for a minute, that I cannot use a ListProperty(). Based on my experiments to date, I can assign only a single entity to 'neighbors' - and I cannot use the "virtual" collection (node_set) to access the list of Node neighbors.

So... my questions are:

1. Does SelfReferenceProperty limit you to a single entity that you can reference?
2. If I instead use a ListProperty, I believe I am limited to 5,000 keys, which I need to exceed.

Thoughts?

Thanks, John

Answer 1

Re 1, yes (if I understand correctly what you're asking): for each property in a model, each instance of the model (AKA "entity") has one value for that property. So e.g. an IntegerProperty has one integer value for a given entity, a SelfReferenceProperty has one value (internally a key string) for a given entity, and so forth. Not a single value for all instances of the module, of course, but I don't believe that's what you mean.

Re 2, the 5000 limit (wow, that's a lot of neighbors for a single node graph that only has 10000 nodes in all!) I believe you're referring to is on indices for an entity -- this article explains the limitation better.

One possible workaround is to have no neighbors property at all on the nodes, but rather a separate Connection model to represent the connection from one node to another (with ReferenceProperty properties, in the Connection, that each refer to a Node). You'll need something extra in the Connection (e.g., progressive counts of incoming and outgoing connections for the affected nodes) to be able to fetch many thousands of them, of course (in multiple queries, given the 1000-at-a-time limitation).

Answer 2

Hi Alex,

Thanks for your reply.

Based on how ReferenceProperty works, I assumed that SelfReferenceProperty would work similarly, providing a virtual collection of referenced elements. It does not seem to work this way. The SelfReference appears to hold a single entity (per "parent" entity instance).

My application is more complex than the example I gave. Specifically, I have the need for a single entity to have a large number of entities on the many side of the relationship. It's a classic one-to-many problem; however, the "many" side in my application can grow to hundreds of thousands. I guess my best (only?) option is to model the relationship as you suggest - with what, in essence, becomes a "link entity."

I will review the limitations of the ListProperty vis-a-vis exploding indexes. I suspect that I will need to take a second look at my data model to work within the app engine's constraints, if at all possible.

BTW, is there still a 1000 at-a-time-limit on queries? I think I recall a video where B. Slatkin mentioned that this was removed --- but I could be mistaken.

Thanks again, John

Answer 3

Update: All wrapped up in my shorts as I was, I forgot the *ReferenceReleationship is "backwards" --- the "list" of things points back to the one side of the relationship. The SelfReferenceProperty works as I thought it would. My bad. I'll still need to deal with the large "many" side of the relationship - and I don't think a ListProperty will do the trick, but I'll post what I find on this forum.