Multiple Uploads to Amazon S3 from Ruby on Rails - What Background Processing System to Use?

Question

I'm developing a Ruby on Rails application that needs to allow the user to simultaneously upload 16 high-quality images at once.

This often means somewhere around 10-20 megabytes (sometimes more), but it's the number of connections that are becoming the most pertinent issue.

The images are being sent to Amazon S3 from Paperclip, which unfortunately opens and closes a new connection for each of the 16 files. Needless to say, I need to move the system to run as background processes to keep my web server from locking up like it already is with no traffic.

My question is, out of all the Rails-based systems to use for background jobs (Starling, BackgroundRb, Spawn, etc.), if there is one that might fit the bill for this scenario better than the others (I'm new to building an in-the-background system anyway, so all of the available systems are equally new to me)?

Answer 1

There's no shortage of rails plugins to do async processing, and basically all of them work fine. Personally I like Delayed Job's api best.

I wouldn't use Starling or other actual queue daemons since for this task using the database to store any necessary state should be just fine.

Answer 2

This might help!

http://aaronvb.com/blog/2009/7/19/paperclip-amazon-s3-background-upload-using-starling-and-workling

EDIT:

It's not possible, through a normal html multipart form, to send files to the background. They have to be done through that request. If you are looking for a way around that, you can try SWFUpload and then once that's done use a background process to handle the Amazon S3 uploads.

Answer 3

You might be interested in my blog post here:

http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip

Answer 4

this is also a good survey blog post http://4loc.wordpress.com/2010/03/10/background-jobs-in-ruby-on-rails/