Using delayed_job across multiple machines with some tasks that need to run locally

Question

I'm trying to use delayed_job for various parts of a rails app. The problem is that if we have multiple instances of the app running, but certain jobs (processing uploads, for example) need to be run by local workers, whereas others could benefit by being run by any worker.

Does anyone have any recommendations for a good approach to having local/non-local job types?

Answer 1

Architecturally, it might make more sense to take the steps so that all jobs can be run from any client.

You mentioned processing uploads, could you share out the directory so that it could be accessed by any of the clients?

Answer 2

I just read "Background Processing with delayed_job" in the latest issue of Rails Magazine and it occurred to me that you could abuse the built-in job priority system.

You can specify a minimum and maximum priority for your workers. Now if your special local-only jobs have a priority of 42 this worker will only process those jobs...

rake jobs:work RAILS_ENV=production MIN_PRIORITY=42 MAX_PRIORITY=42

whilst this worker will process everything except those special local-only jobs:

rake jobs:work RAILS_ENV=production MIN_PRIORITY=0 MAX_PRIORITY=10

This should be flexible enough to achieve what you need. However, I freely admit that I only learnt about this feature today and haven't tried it out myself so YMMV!