Mysql - wondering about scaling a twitter-like application ?

Question

Hi,

I'm developing an app that is vaguely similar to twitter, in that it allows users to follow one another. I wanted to do this using google app engine, for its scalability promises, but it's proving kind of difficult to get running for a few different reasons.

If I were doing this using mysql, I'd have a _users table, and a _followers table. Users go into the users table, follower relationships go into _followers. The problem is that each row in the users table will probably have like 100 corresponding records in the _followers table as users start following one another. So the number of rows is going to explode quickly.

Using app engine, the volume [shouldn't] be a problem.

If I go with mysql, and I do actually start to get some traction, how do I scale this up? Am I going to just end up moving to a distributed database in the end anyway? Should I fight it out with google app engine? I read that Twitter was using mysql, and they've run into this problem, and are now switching to cassandra.

Thanks

Answer 1

Am I going to just end up moving to a distributed database in the end anyway?

If you become as big as e.g. twitter: Yes - or atleast some form of distributed cache.

Should I fight it out with google app engine?

Start making something useful. You're not Twitter - you do not have the experience the people building twitter/google/yahoo has accumulated during scaling a big real world app.

If you start out building something that will scale to the whole world, you will never get it done and you'll get it wrong anyway. If whatever you're making isn't done, there's 0 users and it doesn't matter how well it scales.

Answer 2

GAE doesn't have MySQL (or any RDBMS).

If you can't even read the GAE documentation, you can't build anything remotely close to Twitter's scale.