Keeping a rake job running

Question

Hi,

I'm using delayed_job to run jobs, with new jobs being added every minute by a cronjob.

Currently I have an issue where the rake jobs:work task, currently started with 'nohup rake jobs:work &' manually, is randomly exiting.

While God seems to be a solution to some people, the extra memory overhead is rather annoying and I'd prefer a simpler solution that can be restarted by the deployment script (Capistrano).

Is there some bash/Ruby magic to make this happen, or am I destined to run a monitoring service on my server with some horrid hacks to allow the unprivelaged account the site deploys to the ability to restart it?

Answer 1

The delayed_job docs suggest that you use a monitoring service to manage the rake worker job(s). I use runit--works well.

(You can install it in the mode where it does not replace init.)

Added:

Re: restart by Capistrano: yes, runit enables that. Just do a

sudo sv kill delayed_job

in your Capistrano recipe to kill the delayed_job worker. Runit will then restart it with your newly deployed code base.

Answer 2

I'm running delayed_job on several systems and using the 'daemons' gem to manage. Daemons has a nice option for monitoring what is running and restarting the process if it crashes. This happens once in a while to me and I've not had any trouble with restarts. It's not as good a solution as God or Monit, but I'm in a situation where I can't install either.

An added benefit to using daemons is that the scripts work nicely with Capistrano.

I used this article as a starting point, and modified the scripts to fit my needs:

http://livollmers.net/index.php/2008/11/05/asynchronous-mail-with-delayedjob-god-daemons/

I should note that one production system I have delayed_job running on handles about a million jobs per month, and that's with two delayed_job workers. It's a very robust priority queue and I highly recommend it.

Answer 3

For me the daemons gem was unreliable with delayed_job. Could be a poorly written script (was using the one on collectiveidea's delayed_job github page), and not daemons fault, I'm not really sure. But for whatever reason, it would restart inconsistently on deployments.

I read somewhere this was due to it not waiting for the process to actually exit, so the pid files would get overwritten or something. But I didn't really bother to investigate. I switched to the daemons-spawn gem using these instructions and it seems to be much more reliable now.