background task with multiple workers sharing one rails instance in memory

Question

I need to query mulitple apis to get certain data for each user. I need to do this as a background task. Now there are a lot of users. And ideally I would like all this to happen in parallel. So I could insert a job of each user/api combination. And say have 4 delayed_job workers process all the jobs.

The problem I am facing with this, each DJ worker loads up the entire Rails instance and is consuming 50Mb of memory. Thats 200Mb straight away gone to background workers. Why does each worker need its own copy of my Rails app? Why cant they all share one copy? Why cant they share the copy that passenger uses?

I am currently using Ruby 1.9.1, but I can switch to REE.

My main question is this:

Is there any background task gem/plugin for rails that can spawn multiple workers and yet consume as less memory as possible?

Answer 1

You'll need REE to do it (since it's copy-on-write friendly, which MRI is not), but you could easily achieve this by starting up a spawner process with script/runner and then forking off DelayedJob workers. That would let you share the rails environment and most of the framework memory usage between the DJ workers.