How do I gracefully shut down a Mongrel web server

Question

My RubyOnRails app is set up with the usual pack of mongrels behind Apache configuration. We've noticed that our Mongrel web server memory usage can grow quite large on certain operations and we'd really like to be able to dynamically do a graceful restart of selected Mongrel processes at any time.

However, for reasons I won't go into here it can sometimes be very important that we don't interrupt a Mongrel while it is servicing a request, so I assume a simple process kill isn't the answer.

Ideally, I want to send the Mongrel a signal that says "finish whatever you're doing and then quit before accepting any more connections".

Is there a standard technique or best practice for this?

Answer 1

Try using:

mongrel_cluster_ctl stop

You can also use:

mongrel_cluster_ctl restart

Answer 2

Better question is how to keep your app from consuming so much memory that it requires you to reboot mongrels from time to time.

www.modrails.com reduced our memory footprint significantly

Answer 3

I've done a little more investigation into the Mongrel source and it turns out that Mongrel installs a signal handler to catch an standard process kill (TERM) and do a graceful shutdown, so I don't need a special procedure after all.

You can see this working from the log output you get when killing a Mongrel while it's processing a request. For example:

** TERM signal received.
Thu Aug 28 00:52:35 +0000 2008: Reaping 2 threads for slow workers because of 'shutdown'
Waiting for 2 requests to finish, could take 60 seconds.Thu Aug 28 00:52:41 +0000 2008: Reaping 2 threads for slow workers because of 'shutdown'
Waiting for 2 requests to finish, could take 60 seconds.Thu Aug 28 00:52:43 +0000 2008 (13051) Rendering layoutfalsecontent_typetext/htmlactionindex within layouts/application

Answer 4

Look at using monit. You can dynamically restart mongrel based on memory or CPU usage. Here's a line from a config file that I wrote for a client of mine.

check process mongrel-8000 with pidfile /var/www/apps/fooapp/current/tmp/pids/mongrel.8000.pid
    start program = "/usr/local/bin/mongrel_rails cluster::start --only 8000"
    stop program = "/usr/local/bin/mongrel_rails cluster::stop --only 8000"

    if totalmem is greater than 150.0 MB for 5 cycles then restart       # eating up memory?
    if cpu is greater than 50% for 8 cycles then alert                  # send an email to admin
    if cpu is greater than 80% for 5 cycles then restart                # hung process?
    if loadavg(5min) greater than 10 for 3 cycles then restart          # bad, bad, bad
    if 3 restarts within 5 cycles then timeout                         # something is wrong, call the sys-admin

    if failed host 192.168.106.53 port 8000 protocol http request /monit_stub
        with timeout 10 seconds
        then restart
    group mongrel

You'd then repeat this configuration for all of your mongrel cluster instances. The monit_stub line is just an empty file that monit tries to download. If it can't, it tries to restart the instance as well.

Note: the resource monitoring seems not to work on OS X with the Darwin kernel.

Answer 5

got a question

what happens when /usr/local/bin/mongrel_rails cluster::start --only 8000 is triggered ?

are all of the requests served by this particular process, to their end ? or are they aborted ?

I curious if this whole start/restart thing can be done without affecting the end users...

Answer 6

Boggy:

If you have one process running, it will gracefully shut down (service all the requests in its queue which should only be 1 if you are using proper load balancing). The problem is you can't start the new server until the old one dies, so your users will queue up in the load balancer. What I've found successful is a 'cascade' or rolling restart of the mongrels. Instead of stopping them all and starting them all (therefore queuing requests until the one mongrel is done, stopped, restarted and accepting connections), you can stop then start each mongrel sequentially, blocking the call to restart the next mongrel until the previous one is back up (use a real HTTP check to a /status controller). As your mongrels roll, only one at a time is down and you are serving across two code bases - if you can't do this you should throw up a maintenance page for a minute. You should be able to automate this with capistrano or whatever your deploy tool is.

So I have 3 tasks: cap:deploy - which does the traditional restart all at the same time method with a hook that puts up a maintenance page and then takes it down after an HTTP check. cap:deploy:rolling - which does this cascade across the machine (I pull from a iClassify to know how many mongrels are on the given machine) without a maintenance page. cap deploy:migrations - which does maintenance page + migrations since its usually a bad idea to run migrations 'live'.