Scheduling Tasks for Multiple Rails Apps Without Loading Multiple Environments

Question

On our servers we have deployed multiple Rails applications using Phusion Passenger. Each application has a set of tasks that must be scheduled and run periodically (the usual clear caches, send mail, etc.)

All the examples I've seen in the wild usually show a separate daemon that handles the scheduled tasks for an application. However, this requires that the daemon load the app's environment. While this works when you have one or two apps deployed, it does not scale when there might be 100 apps deployed on a server. (For example, each "daemon" loads up an entire Rails environment plus the app code, which could be well over 50MB. Deploy 100 apps, and you're chewing up 5GB just in scheduler daemons.)

Has anyone solved this problem?

Answer 1

I think that the answer depends on your needs. If those tasks take quite a lot of time, then the simplest solution is to do it in traditional way with daemon. If you want to rely on Rails stack then you must load it to memory in order to use it - you won't escape it.

If your tasks don't need much time, then you can do something like this: use methods that are availble (like delayed_job) but start and stop daemon periodicaly (see here for some tips). So you can start 10 daemons at 9 a.m. and stop them at 10 a.m. Then you can start another 10 and so on. Starting and stopping can be done with cron or any similar tool.

However if your tasks are really simple and very similar in each application, then you can just write your scripts to do the job. On example if "usual clear cache" means that you delete some files from some directories, then just put it to script and run in periodicaly. Sending email can also be done through a script (ruby program). It is really easy to fetch any data from db in ruby. So this program can periodicaly check something like mails_queue table in databases and send emails. If your applications are similar then it will be quite simple to achieve.

Maybe there are some ready solutions to this, but I haven't heard about them.

Answer 2

I had similar issue to solve. I found these solutions available:

1. run tasks initialized by Cron in new process
   • cons: startup time for new instance
2. run tasks in long living daemon
   • cons: daemon occupying memory
3. run tasks in thread inside existing instance
   • cons: passenger spawns X instances and it is hard to ensure that only 1 thread in 1 instance will be run.
   • cons: invoking anything outside request-response cycle may create leaks or instability in passenger.
4. run tasks initialized by HTTP request to existing instance
   • pro: no startup time, uses existing instance, ensures single execution, safe for passanger lifecycle
   • cons: locks one app instance for time of task execution - this may be not an issue if app is thread-safe but needs investigation, as thread-safety is quite new thing in Rails.

Finally I decided to go with option 4. Using cron and curl (or wget), and own CronController accessible only to localhost.