Has anybody done a comparison of the overhead of the various background processing techniques?
Background/RB, Starling, Workling MemcacheQ Beanstalk Background Job (Bj) delayed_job (Dj)
I will be implementing one of them on a slice and would like to know how much memory they take up so I can factor it into my decision making.
I would also be interested in a comprehensive comparison, but one thing I can say is that BackgroundRB is considered deprecated by its author. At EngineYard they are specifically recommending BackgroundJob after having had intractable problems with BackgroundRB. I've heard nothing about the other options you mention however.
For low maintenance I like Background Job. It runs in your Rails process or via cron so there are no daemon processes to monitor. On my server Bj is currently using 35636 RSS (approximately one Rails process worth).
I'm always surprised when I hear about people using BackgrounDRB because it is basically unmaintained.
It is going to vary depending on your Rails application itself.
Most of these processors depend on your Rails objects will essentially load the entire Rails instance into memory. Your App memory will depend on the number of models, the impact of any plugins and the prevailing climactic conditions of your environment.
I had a 256mb Slice running several Mongrels and BackgroundRB and found that the background process used about the same memory as a Mongrel instance.
One option I have always liked is putting your scheduled logic in a Controller and calling this via Cron using wget or Curl. You get to leverage the existing Rails application and there is very little overhead in setting it up. The only reason I didn't go with this option in the above case was a requirement for hitting the queue every 5 seconds (Cron can only run every minute).