views:

146

answers:

3

I have a ruby application that's used by users via a rails web app.

The use case that i need to solve is: each user needs to compile create an entity on database via the web page and put a really big attachment (can be 100gigs or more too) related to the entity.

Now this is too big to be handled by normal upload plugins, i am searching for a way to solve this problem, my ideas are: - delegate the upload via http to backgroundrb - develop a GTK gui and ship it to user to do the upload

any other hint or ideas?

thanks

+1  A: 

If it's >100GB, you should probably be using an upload manager of some kind. If your network speed is fast enough, you could just build a trusted applet to communicate with the server. The trusted applet would have full file permission, so it would work just like a standalone application. In this manner, you could avoid needing to distribute a separate utility for file transfer.

Alternatively, you could generate some token on the web, and then let users use a standalone application with the given token to upload their file.

Stefan Kendall
Good idea, can you point me on a open source solution already developed and used?
antani
Google points me to jumploader, but I don't know if it's open source. I know that certain bands have utilized similar utilities to transfer lossless audio (>200MB) to end users, so I assume this is similar. Actually transferring a file is easy once you realize that you're just writing a client/server java application, but you'd want to find libraries or a tool to allow for file resuming, I would assume. Jumploader appaers to support this.
Stefan Kendall
http://jumploader.com/demo.htmlThe demo looks promising.
Stefan Kendall
I already found that, but it's not open, and the license costs too much.. And my project want to be open source and free.
antani
Hmm. You could always write the application yourself. It's not that bad, unless you want to do true file resuming. Any easy way to bypass that would be to zip the upload file and split it into managable parts. Then, just compare file sizes of the split files to determine if the whole file was sent, possibly also with an md5 check or crc of some kind. Calling a JFileChooser is trivial, and from there it's just a batch upload with integrity checking at the end.
Stefan Kendall
And hey, if you do that portion well enough, make it open source and free, for the rest of us :).
Stefan Kendall
A: 

Uploading a 100GB file through your webapp will not be very good. That upload is going to occupy your web server for the time it takes to upload the file making it inaccessible to others.

I'd consider using FTP or perhaps even having your users upload directly to S3 instead and then manage the files from there since I assume that you have to store these files somewhere anyways.

Randy Simon
No, S3 can't be a solution because the data are really confidential.And FTP isn't a solution due to other reasons.
antani
What other reasons? Would SFTP or SCP work instead?
Matthew
A: 

Transferring 100GB over a web interface makes me cringe. It would be much better to write/use a standalone application to do the transfer.

If you are already using Ruby, you can write a Ruby-based app that uploads over FTP. For example:

require 'net/ftp'

Net::FTP.open('uploads.yoursite.com','username','password') {|ftp|
    ftp.login('username','password')
    ftp.put 'filename'
    if (ftp.last_response != "266 Transfer complete.\n")
        puts "Error with FTP upload\nResponse was: #{ftp.last_response}"
    end
}

I use this code to upload auto-generated data files to another server that archives them. There are several different libraries for building simple user interfaces in Ruby, and all you would need is a simple window where the user can enter their username and password, select the file to upload, and click the "Go" button.

The Ruby SSH libraries make it possible to do a secure file transfer over SFTP (hint: require 'net/sftp'). I haven't used it myself, but the docs make it look as easy as FTP. IIRC, SFTP has native support for resuming interrupted transfers.

You could also use a utility like WinSCP, which is an open-source tool that can upload using FTP, SFTP, or SCP. For non-Windows systems, there's Cyberduck for OS X and Kasablanka or gFTP for Linux.

bta