modrails - rogue ruby processes consuming 100% cpu

Question

I'm having ruby instances from mod_rails go "rogue" -- these processes are no longer listed in passenger-status and utilize 100% cpu.

Other than installing god/monit to kill the instance, can anyone give me some advice on how to prevent this? I haven't been able to find anything in the logs that helps.

Answer 1

This is a recurring issue with passenger. I've seen this problem many times helping people that ran ruby on rails with passenger. I don't have a fix but you might want to try this http://www.modrails.com/documentation/Users%20guide.html#debugging_frozen

Answer 2

If you're using Linux, you can install the "strace" utility to see what the Ruby process is doing that's consuming all the CPU. That will give you a good low-level view. It should be available in your package manager. Then you can:

$ sudo strace -p 22710
Process 22710 attached - interrupt to quit
...lots of stuff...
(press Ctrl+C)

Then, if you want to stop the process in the middle and dump a stack trace, you can follow the guide on using GDB in Ruby at http://eigenclass.org/hiki.rb?ruby+live+process+introspection, specifically doing:

gdb --pid=(ruby process)
session-ruby
stdout_redirect
(in other terminal) tail -f /tmp/ruby_debug.(pid)
eval "caller"

You can also use the ruby-debug Gem to remotely connect to debug sockets you open up, described in http://duckpunching.com/passenger-mod_rails-for-development-now-with-debugger

There also seems to be a project on Github concerned with debugging Passenger instances that looks interesting, but the documentation is lacking: http://github.com/ddollar/socket-debugger/tree/master

Answer 3

We saw something similar to this with very long running SQL queries.

MySQL would kill the queries because they exceeded the long running limit and the thread never realized that the query was dead.

You may want to check the database logs.

Answer 4

I'm having the exact same problem running merb+passenger. Every once in a while a merb-process hangs and starts using 50-100% cpu. Did not happen before I upgraded to 2.1.2 (from 2.0.6).

Is there an easy way to monitor this with God? God does not start the procsesses, so I cannot find an easy way to do it, except write a custom condition for apache. But I only want to kill the processes gone astray, not relaunch apache every time.