Paperclip to upload images to S3 in Rails. The files upload at a very slow rate. A work-around?

Question

I'm working on a rails app where the user will be uploading large quantities of images.

My current setup: Using SWFUpload to upload multiple files at once using the Paperclip plugin with S3 storage. After the original image is uploaded to S3, Delayed_Job is used for the post processing (thumbnails, etc).

The problem I have is that the images upload at a very slow rate. I'm assuming the default Paperclip setup is that the image will go from the user to -> my server to -> s3.

I was thinking that I can have the images upload directly to s3 but I'm not sure how to implement that with Paperclip and post processing. I couldn't find any plugins or examples dealing with this.

Does anyone have suggestions? If not, can you point me in the right direction?

Thanks in advance!

Tim

Answer 1

I've ran into this same problem a few times. The way I solved it was by creating 2 models, a Image model and a TempImage model, which inherits from the Image model. This requires you to have a type column on your Image table. The TempImage model saves the image locally, then when you access it from the Image model directly and resave it, it will follow whatever is defined in the Image model, being Amazon S3.

Example:

# Will save in the database as a TempImage inside the Image table
temp = TempImage.create(:asset => File.new('some_path', 'r'))

# When you find it again through the Image model, it bypasses the type column
# so next time you save it, it is saved as an Image.
amazon = Image.find(temp.id)
amazon.save!

Here is my delayed job:

class MoveToS3Job < Struct.new(:temp_revision_id)
  def perform
    upload = Image.find(temp_revision_id)
    temp_path = File.expand_path("tmp/uploads/#{upload.asset_file_name}", Rails.root)
    upload.asset = File.new(temp_path, 'r')
    upload.save!

    if File.exists?(temp_path) && !File.directory?(temp_path)
      File.delete(temp_path)
    end
  rescue ActiveRecord::RecordNotFound
    # If the record wasn't found, do some sort of
    # error report, but don't keep it in the queue.
  end
end

Here is the TempImage model:

class TempImage < Image
  has_attached_file :asset, {
    :path => ":rails_root/tmp/uploads/:basename_:updated_at.:extension"
  }
end

Then the original Image model:

class Image < ActiveRecord::Base
  # Validations
  validates :asset, :presence => true

  # Paperclip
  has_attached_file :asset, :styles => {
    :preview => ['100x100#', :png],
    :thumb => ['50x50#', :png]
  },
  :default_style => :thumb,
  :storage => :s3,
  :bucket => 'bucket-name',
  :s3_credentials => File.expand_path('config/s3.yml', Rails.root),
  :path => "photos/:id_partition/:style.:extension"
end

Your original Image model should always contain your post processing, as that will be done in the background.

You can always overwrite some methods to make it a little cleaner, but this gives you a better idea of how it works and what you need to do to so you can have it work like you want it to.