MIgrating data from EC2 to S3?

Question

I am running a Rails app using Paperclip to take care of file attachments and image resizing, etc. The app is currently hosted on EngineYard cloud, and all attachments are stored in their EBS. Thinking about using S3 to handle all Paperclip attachments.

Does anyone know of a good and safe way for this migration? many thanks!

Answer 1

You could work up a rake task that iterates over your attachments and pushes each to S3. I used this one awhile back with attachment_fu -- wouldn't be too different. This uses the aws-s3 gem.

Basically the process is: 1. Select files from the database that need to be moved 2. Push them to S3 3. Update database to reflect that the file is no longer stored locally (this way you can do them in batches and don't need to worry about pushing the same file twice).

@attachments = Attachment.stored_locally
@attachments.each do |attachment|

  base_path = RAILS_ROOT + '/public/assets/'
  attachment_folder = ((attachment.respond_to?(:parent_id) && attachment.parent_id) || attachment.id).to_s
  full_filename = File.join(base_path, ("%08d" % attachment_folder).scan(/..../), attachment.filename)
  require 'aws/s3'

  AWS::S3::Base.establish_connection!(
    :access_key_id        => S3_CONFIG[:access_key_id],
    :secret_access_key    => S3_CONFIG[:secret_access_key]
  )

  AWS::S3::S3Object.store(
    'assets/' + attachment_folder + '/' + attachment.filename,
    File.open(full_filename),
    S3_CONFIG[:bucket_name],
    :content_type => attachment.content_type,
    :access => :private
  )

  if AWS::S3::Service.response.success?
    # Update the database
    attachment.update_attribute(:stored_on_s3, true)

    # Remove the file on the local filesystem
    FileUtils.rm full_filename

    # Remove directory also if it is now empty
    Dir.rmdir(File.dirname(full_filename)) if (Dir.entries(File.dirname(full_filename))-['.','..']).empty?
  else
    puts "There was a problem uploading " + full_filename
  end
end

Answer 2

I found myself in the same situation and took bensie's code and made it work for myself - this is what I came up with:

require 'aws/s3'

# Ensure you do the following:
#   export AMAZON_ACCESS_KEY_ID='your-access-key'
#   export AMAZON_SECRET_ACCESS_KEY='your-secret-word-thingy'
AWS::S3::Base.establish_connection!


@failed = []
@attachments = Asset.all # Asset paperclip attachment is: has_attached_file :attachment....
@attachments.each do |asset|
  begin
    puts "Processing #{asset.id}"
    base_path = RAILS_ROOT + '/public/'
    attachment_folder = ((asset.respond_to?(:parent_id) && asset.parent_id) || asset.id).to_s
    styles = asset.attachment.styles.keys
    styles << :original
    styles.each do |style|
      full_filename = File.join(base_path, asset.attachment.url(style, false))


      AWS::S3::S3Object.store(
        'attachments/' + attachment_folder + '/' + style.to_s + "/" + asset.attachment_file_name,
        File.open(full_filename),
        "swellnet-assets",
        :content_type => asset.attachment_content_type,
        :access => (style == :original ? :private : :public_read)
      )

      if AWS::S3::Service.response.success?        
        puts "Stored #{asset.id}[#{style.to_s}] on S3..."
      else
        puts "There was a problem uploading " + full_filename
      end
    end
  rescue
    puts "Error with #{asset.id}"
    @failed << asset.id
  end
end

puts "Failed uploads: #{@failed.join(", ")}" unless @failed.empty?

Of course, if you have multiple models you will need to adjust as necessary...