Google App Engine - How to implement the activity stream in a social network

Question

I want some ideas on the best practice to implement an activity stream for a social network im building in app engine (PYTHON)

I first want to keep a log for all activities of each user - so that we have a history. i.e. someone became a friend, added a picture, changed their address etc. This way we have a users history available should we need it. Also mean we can remove friendship joins, change user data but have a historical log.

I also want to stream a users activity to their friends. for this only the last X activities need to be kept - that is in the scenario that messages are sent to friends when an activity occurs.

Its pretty straight forward designing a history log - ie: when, what, where. The complication comes as to how we notify friends of a user as to their activity.

In our app friendships are not mutual - ie they are based on the twitter following model. Some accounts could have thousands of followers.

What is the best approach to model this.

1. using a many to many join table and doing a costly query -
2. using a feed class that fired a copy of the activity to all the subscribers - maybe into mcache? As their maybe a need to fire thousands of messages i would imagine a cron job would need to be used.

Any help ideas thoughts on this

Thx

Answer 1

1. Don't do joins. They're too expensive, you'll burn through your quota in no time.
2. You can use a task queue, it's a bit like a cron job (i.e. stuff happens outside of the original request) but you can start them at will. memcache would be good if you're ok with loosing some activity at times the cache is flushed...

Answer 2

There's a great talk by Brett Slatkin called Building Scalable, Complex Apps on App Engine from last year's Google I/O, in which the example is a Twitter-like application, where users' updates are pushed to their followers. Basically exactly what you're trying to do.

I highly recommend the video for anyone writing an App Engine app, it's really helpful.