Passenger hosted Rails app *painfully* slow, but the server is a beast...

Question

I have been working to deploy a relatively large Rails app (Rails 2.3.5) and recently doing some load testing we discovered that the throughput for the site is way below the expected level of traffic.

We were running on a standard 32bit server, 3GB of RAM with Centos, and we were running Ruby Enterprise Edition (latest build), Passenger (Latest build) and Nginx (Latest build) - when there is only one or two users the site runs fine (as you would expect) however when we try to ramp up the load to ~50 concurrent requests it completely dies. (Apache Bench report ~2.3 req/sec, which is terrible)

We are running RPM and trying to determine where the load issue is, but it's pretty evenly distributed across Rails, SQL and Memcached, so we're more or less going through and optimizing the codebase.

Out of sheer desperation we spun up a large EC2 instance (Ubuntu 9.10, 7.5GB RAM, 2 Compute Units/Cores) and setup the same configuration as the original server, and while there are more resources we were still seeing pathetic results.

So, after spending too much time trying to optimize, playing with caching configuration etc I decided to test the throughput of some mongrels, and ta-da, they are performing much much better then Passenger.

Currently the configuration is 15x Mongrels being proxied via Nginx, and we seem to be meeting our load requirements just but it's not quite enough to make me comfortable with going live... What I was wondering is if anyone knows of some possible causes for this...?

My configuration for passenger/nginx was:

• Nginx workers: tried between 1 and 10, usually three though.
• Passenger max pool size: 10 - 30 (yes, these numbers are quite high)
• Passenger global queueing: tried both on and off.
• NGinx GZip on: yes

It might pay to note that we had increased the nginx max client body size to 200m to allow for large file uploads.

Anyway suggestions would be really appreciated, while the mongrels are working fine it changes how we do things a lot and I would really prefer to use Passenger - besides, wasn't it supposed to make this easier and perform better?

Answer 1

As a first step I would deploy a minimal "Hello World" type Rails application to your environment and see what throughput you get with that. Doing that will at least tell you if your problem is with the environment or somewhere in your application.

Answer 2

Maybe your sql pool size is too small? This essentially limits the parallelism of database workloads in your application which in turn builds up to much increased load as soon as you put work on your app stack...