Scaling with a cluster- best strategy

Question

I am thinking about the best strategy to scale with a cluster of servers. I know there is no hard and fast rules, but I am curious what people think about these scenarios:

1. cluster of combination app/db servers that are round robin (with failover) balanced using dnsmadeeasy. the db's are synced using replication. Has the advantage that capacity can be augmented easily by adding another server to the cluster, and it is naturally failsafe.

2. cluster of app servers, again round robin load balanced (with failover) using dnsmadeeasy, all reporting to a big DB server in the back. easy to add app servers, but the single db server creates a single failure point. Could possible add a hot standby with replication.

3. cluster of app servers (as above) using two databases, one handling reads only, and one handling writes only.

Also, if you have additional ideas, please make suggestions. The data is mostly denormalized and non relational, and the DBs are 50/50 read-write.

Answer 1

Take 2 phisical machines and make them Xen servers

• A. Xen Base alpha
• B. Xen Base beta

In each one do three virtual machines:

1. "web" server for statics(css,jpg,js...) + load balanced proxy for dynamic request (apache+mod-proxy-balancer,nginx+fair)
2. "app" server (mongrel,thin,passenger) for dynamic requests
3. "db" server (mySQL, PostgreSQL...)

Then your distribution of functions can be like this:

• A1 owns your public ip and handle requests to A2 and B2
• B1 pings A1 and takes over if ping fails
• A2 and B2 take dynamic request querying A3 for data
• A3 is your dedicated data server
• B3 backups A3 second to second and offer readonly access to make copies, backups etc. B3 pings A3 and become master if A3 becomes unreachable

Hope this can help you some way, or at least give you some ideas.

Answer 2

It really depends on your application.

I've spent a bit of time with various techniques for my company and what we've settled on (for now) is to run a reverse proxy/loadbalancer in front of a cluster of web servers that all point to a single master DB. Ideally, we'd like a solution where the DB is setup in a master/slave config and we can promote the slave to master if there are any issues. So option 2, but with a slave DB. Also for high availability, two reverse proxies that are DNS round robin would be good. I recommend using a load balancer that has a "fair" algorithm instead of simple round robin; you will get better throughput. There are even solutions to load balance your DB but those can get somewhat complicated and I would avoid them until you need it.

Rightscale has some good documentation about this sort of stuff available here: http://wiki.rightscale.com/ They provide these types of services for the cloud hosting solutions.

Particularly useful I think are these two entries with the pictures to give you a nice visual representation.

The "simple" setup:
http://wiki.rightscale.com/1._Tutorials/02-AWS/02-Website_Edition/2._Deployment_Setup

The "advanced" setup:
http://wiki.rightscale.com/1._Tutorials/02-AWS/02-Website_Edition/How_do_I_set_up_Autoscaling%3f

Answer 3

I'm only going to comment on the database side:

With a normal RDBMS a 50/50 read/write load for the DB will make replication "expensive" in terms of overhead. For almost all cases having a simple failover solution is less costly than implementing a replicating active/active DB setup. Both in terms of administration/maintenance and licensing cost (if applicable).

Since your data is "mostly denormalized and non relational" you could take a look at HBase which is an OSS implementation of Google Bigtable, a column based key/value database system. HBase again is built on top of Hadoop which is an OSS implementation of Google GFS.

Which solution to go with depends on your expected capacity growth where Hadoop is meant to scale to potentially 1000s of nodes, but should run on a lot less as well.

I've managed active/active replicated DBs, single-write/many-read DBs and simple failover clusters. Going beyond a simple failover cluster opens up a new dimension of potential issues you'll never see in a failover setup.

If you are going for a traditional SQL RDBMS I would suggest a relatively "big iron" server with lots of memory and make it a failover cluster. If your write ratio shrinks you could go with a failover write cluster and a farm of read-only servers.

The answer lies in the details. Is your application CPU or I/O bound? Will you require terabytes of storage or only a few GB?