How to upload big files to a rails application?

Question

I have a ruby application that's used by users via a rails web app.

The use case that i need to solve is: each user needs to compile create an entity on database via the web page and put a really big attachment (can be 100gigs or more too) related to the entity.

Now this is too big to be handled by normal upload plugins, i am searching for a way to solve this problem, my ideas are: - delegate the upload via http to backgroundrb - develop a GTK gui and ship it to user to do the upload

any other hint or ideas?

thanks

Answer 1

If it's >100GB, you should probably be using an upload manager of some kind. If your network speed is fast enough, you could just build a trusted applet to communicate with the server. The trusted applet would have full file permission, so it would work just like a standalone application. In this manner, you could avoid needing to distribute a separate utility for file transfer.

Alternatively, you could generate some token on the web, and then let users use a standalone application with the given token to upload their file.

Answer 2

Uploading a 100GB file through your webapp will not be very good. That upload is going to occupy your web server for the time it takes to upload the file making it inaccessible to others.

I'd consider using FTP or perhaps even having your users upload directly to S3 instead and then manage the files from there since I assume that you have to store these files somewhere anyways.

Answer 3

Transferring 100GB over a web interface makes me cringe. It would be much better to write/use a standalone application to do the transfer.

If you are already using Ruby, you can write a Ruby-based app that uploads over FTP. For example:

require 'net/ftp'

Net::FTP.open('uploads.yoursite.com','username','password') {|ftp|
    ftp.login('username','password')
    ftp.put 'filename'
    if (ftp.last_response != "266 Transfer complete.\n")
        puts "Error with FTP upload\nResponse was: #{ftp.last_response}"
    end
}

I use this code to upload auto-generated data files to another server that archives them. There are several different libraries for building simple user interfaces in Ruby, and all you would need is a simple window where the user can enter their username and password, select the file to upload, and click the "Go" button.

The Ruby SSH libraries make it possible to do a secure file transfer over SFTP (hint: require 'net/sftp'). I haven't used it myself, but the docs make it look as easy as FTP. IIRC, SFTP has native support for resuming interrupted transfers.

You could also use a utility like WinSCP, which is an open-source tool that can upload using FTP, SFTP, or SCP. For non-Windows systems, there's Cyberduck for OS X and Kasablanka or gFTP for Linux.