How to prepare to be tech crunched

Question

There is a good chance that we will be tech crunched in the next few days. Unfortunately, we have not gone live yet so we don't have a good estimation of how our system handles a production audience.

Our production setup consists of 2 EngineYard slices each with 3 mongrel instances, using Postgres as the database server.

Obviously a huge portion of how our app will hold up is to do with our actual code and queries etc. However, it would be good to see if there are any tips/pointers on what kind of load to expect or experiences from people who have been through it. Does 6 mongrel instances (possibly 8 if the servers can take it) sound like it will handle the load, or are at least most of it?

Answer 1

Look into some load testing software like WEBLoad or if you have money, Quick Test Pro. This will help give you some idea. WEBLoad might be the best test in your situation.

You can generate thousands of virtual nodes hitting your site and you can inspect the performance of your servers from that load.

Answer 2

In my experience having watched some of our customers absorb a crunching, the traffic was fairly modest- not the bone crushing spike people seem to expect. Now, if you get syndicated and make on Yahoo's page or something, things may be different.

Search for the experiences of Facestat.com if you want to read about how they handled it (the Yahoo FP.)

My advise is just be prepared to turn off signups or go to a more static version of your site if your servers get too hot. Using a monitoring/profiling tool is a good idea as well, I like FiveRuns Manage tool for ease of setup.

Answer 3

Since you're using EngineYard, you should be able to allocate more machines to handle the load if necessary

Answer 4

Your big problems will probably not be the number of incoming requests, but will be the amount of data in your database showing you where your queries aren't using the indexes your expecting, or are returning too much data, e.g. The User List page works with 10 users, but dies when you try to show 10,000 users on that one page because you didn't add pagination (will_paginate plugin is almost your friend - watch out for 'select count(*)' queries that are generated for you)

So the two things to watch:

1. Missing indexes
2. Too much data per page

For #1, there's a plugin that runs an 'explain ...' query after every query so you can check index usage manually

There is a plugin that can generate data for you for various types of data that may help you fill your database up to test these queries too.

For #2, use will_paginate plugin or some other way to reduce data per page.

Answer 5

We've got basically the same setup as you, 2 prod slices and a staging slice at EY. We found ab to be a great load testing tool - just write a bash script with the urls that you expect to get hit and point it at your slice. Watch NewRelic stats and it should give you some idea of the load your app can handle and where you might need to optimise.

We also found query_reviewer to be very useful as well. It is great for finding those un-indexed tables and n+1 queries.

Answer 6

I have worked on several rails applications that experienced high load due to viral growth on Facebook.

Your mongrel count should be based on several factors. If your mongrels make API calls or deliver email and must wait for responses, then you should run as many as possible. Otherwise, try to maintain one mongrel per CPU core, with maybe a couple extra left over.

Make sure your server is using a Fair Proxy Balancer (not round robin). Here is the nginx module that does this: http://github.com/gnosek/nginx-upstream-fair/tree/master

And here are some other tips on improving and benchmarking your application performance to handle the load:

ActiveRecord

The most common problem Rails applications face is poor usage of ActiveRecord objects. It can be quite easy to make 100's of queries when only one is necessary. The easiest way to determine if this could be a problem with your application is to set up New Relic. After making a request to each major page on your site, take a look at the newrelic SQL overview. If you see a large number of very similar queries sequentially (select * from posts where id = 1, select * from posts where id = 2, select * from posts...) this may be a sign that you need to use a :include in one of your ActiveRecord calls.

Some other basic ActiveRecord tips (These are just the ones I can think of off the top of my head):

1. If you're not doing it already, make sure to correctly use indexes on your database tables.

2. Avoid making database calls in views, especially partials, it can be very easy to lose track of how much you are making database queries in views. Push all queries and calculations into your models or controllers.

3. Avoid making queries in iterators. Usually this can be done by using an :include.

4. Avoid having rails build ActiveRecord objects for large datasets as much as possible. When you make a call like Post.find(:all).size, a new class is instantiated for every Post in your database (and it could be a large query too). In this case you would want to use Post.count(:all), which will make a single fast query and return an integer without instantiating any objects.

5. Associations like User..has_many :objects create both a user.objects and user.object_ids method. The latter skips instantiation of ActiveRecord objects and can be much faster. Especially when dealing with large numbers of objects this is a good way to speed things up.

6. Learn and use named_scope whenever possible. It will help you keep your code tiny and makes it much easier to have efficient queries.

External APIs & ActionMailer

As much as you can, do not make API calls to external services while handling a request. Your server will stop executing code until a response is received. Not only will this add to load times, but your mongrel will not be able to handle new requests.

If you absolutely must make external calls during a request, you will need to run as many mongrels as possible since you may run into a situation where many of them are waiting for an API response and not doing anything else. (This is a very common problem when building Facebook applications)

The same applies to sending emails in some cases. If you expect many users to sign up in a short period of time, be sure to benchmark the time it takes for ActionMailer to deliver a message. If it's not almost instantaneous then you should consider storing emails in your database an using a separate script to deliver them.

Tools like BackgroundRB have been created to solve this problem.

Caching

Here's a good guide on the different methods of caching in rails.

Benchmarking (Locating performance problems) If you suspect a method may be slow, try benchmarking it in console. Here's an example:

>> Benchmark.measure { User.find(4).pending_invitations }
=> #<Benchmark::Tms:0x77934b4 @cutime=0.0, @label="", @total=0.0, @stime=0.0, @real=0.00199985504150391, @utime=0.0, @cstime=0.0>

Keep track of methods that are slow in your application. Those are the ones you want to avoid executing frequently. In some cases only the first call will be slow since Rails has a query cache. You can also cache the method yourself using Memoization.

NewRelic will also provide a nice overview of how long methods and SQL calls take to execute.

Good luck!