How to sort by a counter when using sharded counters

Question

I have an application where the main entity is a Story and users can vote for each story. Each vote increments a vote_count for the story.

I am concerned about write contention on the story so I plan to use a sharded counter for each story to track the votes.

Now my question: how could I get a list of stories ordered by number of votes? For example: show the 50 most highly votes stories.

My initial thought is to have a task run periodically that reads the counter values and updates a property on the actual story. It would be OK that the results of the query by vote were slightly out of date.

Answer 1

It sounds like you may be doing a bit of premature optimization. I would skip the sharded counters until it becomes obvious that you need them. If you are pretty sure you will, then by all means, start with them. As for running a periodic task and caching results in a property for each story, that may be another premature optimization.

I have no direct experience with google app engine so hopefully somebody that does will have some info to share.

Answer 2

Periodically adding up data might be a good strategy to counter the sharding dispersion of counters.

You could also try other strategies for counting without shards, as has been described elsewhere:

http://blog.notdot.net/2010/04/High-concurrency-counters-without-sharding

(there you keep your counter in memcache, and periodically flush the accumulated value to the datastore)

How critical is your app to slight counting errors?