Good way of implementing a twitter-like follower system?

Question

Hi,

I'm trying to create a twitter-like follower system (users can follow one another). I'm confused about a good way to store the follower relationships. I'm using JDO (on google app engine).

The first thing that comes to mind is to keep a Set for followers, and the ppl you are following. Something like:

class User {

   private String mUsername;

   private Set<String> mFollowers;

   private Set<String> mFollowees;
}

I'm worried about what happens when those sets grow to have like 10,000+ entries in them. Viewing a user's page is going to be a common operation, and I'd hate to have to load the entire Sets every time my API needs to generate user info. I'm only going to be showing 50 followers at a time anyway, so it makes no sense to load the entire Set.

An alternate could be using an intermediate class to store relationships, this way they are not bound to the User object. Paging should then also be easy (I think). For example, whenever I want to follow a user, I'd create an instance of this object:

class RelationshipInfo {

    private String mMyUsername;

    private String mUsernameYouAreFollowing;
}

so when I view a user's page, I could query for the first 50 such records above given the user's id. Does that make any performance sense? I'm not sure if this is better than the first option above. This way would require more trips to the datastore.

Any thoughts would be great,

Thanks

Answer 1

Brett Slatkin's Building Scalable, Complex Apps on App Engine talk from last year's Google I/O actually uses a Twitter-like application as its example. Even aside from that, it's a great talk and I highly recommend it even if it didn't relate specifically to what you're asking.

Also, you may want to check out Jaiku, an open-source Twitter-like application built on App Engine.