Best practices for withstanding launch day traffic burst

Question

We are working on a website for a client that (for once) is expected to get a fair amount of traffic on day one. There are press releases, people are blogging about it, etc. I am a little concerned that we're going to fall flat on our face on day one. What are the main things you would look at to ensure (in advance without real traffic data) that you can stay standing after a big launch.

Details: This is a L/A/M/PHP stack, using an internally developed MVC framework. This is currently being launched on one server, with Apache and MySQL both on it, but we can break that up if need be. We are already installing memcached and doing as much PHP-level caching as we can think of. Some of the pages are rather query intensive, and we are using Smarty as our template engine. Keep in mind there is no time to change any of these major aspects--this is the just the setup. What sorts of things should we watch out for?

Answer 1

To prepare or handle a spike (or peak) performance, I would first determine whether you are ready through some simple performance testing with something like jmeter.

It is easy to setup and get started and will give you early metrics whether you will handle an expected peak load.

However given your time constraints, other steps to take would be to prepare static versions of content that will attract the highest attention (such as press releases, if your launch day). Also ensure that you are making the best use of client side caching (1 less request to your server can make all the difference). The web already designed for extremely high scalability and effective use content caching is your best friend in these situations.

There is an excellent podcast on high scalability on software engineering radio on the design of the new Guardian website when things calm down.

good luck on the launch

Answer 2

Measure first, then optimize. Have you done any loadtesting? Where are the bottlenecks?

Once you know your bottlenecks then you can intelligently decide if you need additional DB boxes or web boxes, right now you'd just be guessing.

Also, how does your loadtesting results compare against your expected traffic? Can you handle 2x the expected traffic? 5x? How easy/fast can you acquire & release extra hardware? I'm sure the business requirement is to not fail during launch, so make sure you have lots of capacity available, you can always release it afterwards when the load has stabilized and you know what you need.

Answer 3

I'd, personally, do a few things

1) Put in some sort of load balancer/database replication system

This means that you can have your service spread across multiple servers. Can't afford to have more than one server permanently? Use Amazon E3 - It's good for putting in place for things like this (switch on a few more servers to handle the load)

2) Code in some "High Load" restrictions

For example, if your searching is inefficient - switch it off when load gets to a certain level. "Sorry, we're busy, try again later for searching"

3) Load test... Use something like ApacheBench to stress test your servers.

4) Personally, I think that switching "Keep-Alive" Connections off is better. It may slightly reduce overall performance, but - it means that instead of having something where the site works well for a few people, and the others get timeouts, everyone gets inconsistent service, if it gets to that level

Linux Format did a good article on "How to survive a slashdotting"... which I've found useful in the past. It's available online as a PDF

Answer 4

I would at least factor out all static content. Setup another vhost somewhere else and load all the graphics/css/js onto it. You can buy some extra cycles offloading the serving of that type of content. If your really concerned you can signup and use a content distribution service. There are lots now similar to Akamai and quite cheap.

Another idea might be to utilize apache mod_proxy to keep generated page output for a specific amount of time. APC would also be quite usable.. You could employ output buffering capture + the last modified time of related data on the page, and use the APC cached version. If the page isn't valid anymore, you regenerate and store in APC again.

Good luck, it'll be a learning experience!

Answer 5

Have a beta period where you allow in as many users as you can handle, measure your site's performance, work out bugs before you go live.

You can either control the number of users explicitly in a private beta, or a Google-style semi-public beta where each user has a number of referrals that they can offer to their friends.

Answer 6

Look into using Varnish- it's a caching reverse proxy server (like squid, but much more single purpose). I've run some pretty big sites behind it, it seemed to work really well.

Answer 7

Basic First Steps to harden your site for high traffic.

1) Use a low-cost tool like https://browsermob.com/ to load-test your site. At a minimum you should be looking at 100K unique visitors per hour. If you get an ad off of MSN home page, look to be able to handle 500K uniques per hour.

2) Move all static graphic/video content to a CDN. Edgecast and Amazon are two excellent choices.

3) Use Jet Profiler to profile your MySQL Server to analyze any slow performing queries. Minor changes can have huge benefits.