Scaling exercises to practice

Question

I was trying to find online some exercises to practice scaling techniques (memchached, SQL Optimization, sharding dbs), but I could only find descriptions of these techniques, not any project on which to try them.

This link with slides on scaling techniques, is an interesting one, as it sums up some tools to achieve scalability quite well.

Is there a projecteuler kind of site for these kind of activities? Or at least some excercises (such as a downloadable ASP.NET/PHP site with obvious slowdowns, concurrency issues, subtle bugs) for people to try and learn how to fight this issue?

Answer 1

It might be interesting to hack at Wordpress. Their caching plugins take care of a lot of scaling issues but it would be cool to write your own plugin or hack at the source to cut down on SQL queries or to cache static pages. If you come up with something, make sure to let the rest of the community know!

Answer 2

George's slides are definitely a good basis to work from. Note that he is not talking about a specific technique or technology; rather he's discussing more general architectural and design decisions that will help your application scale as a whole.

I personally think this sort of high-level thinking would be much more valuable than individual optimisation techniques. Perhaps you could take a well known web application and hack it until it scales well across multiple machines? A cluster of lots of cheap, low-power EC2 machines could be really useful here. Getting an existing or new application to run properly across a number of machines would be a fantastic exercise.

Counter-intuitively, rather than getting as much as possible to run on a single machine, I'd say it would be much more educational to get the same application running on several machines.

Once you have that, it makes sense to move onto more specific improvements like a separate static content tier, memcached, DB sharding, batch operations and so on.

In terms of specific projects to work on, how about cloning Twitter, Flickr or The Pirate Bay. They've all had performance and scaling challenges in the past.

Answer 3

I find that the site High Scalability has some nice insights.