Best Practice for Uploading Many (2000+) Images to A Server

Question

Hello,

I have a general question about this.

When you have a gallery, sometimes people need to upload 1000's of images at once. Most likely, it would be done through a .zip file. What is the best way to go about uploading this sort of thing to a server. Many times, server have timeouts etc. that need to be accounted for. I am wondering what kinds of things should I be looking out for and what is the best way to handle a large amount of images being uploaded.

I'm guessing that you would allow a user to upload a zip file (assuming the timeout does not effect you), and this zip file is uploaded to a specific directory, lets assume in this case a directory is created for each user in the system. You would then unzip the directory on the server and scan the user's folder for any directories containing .jpg or .png or .gif files (etc.) and then import them into a table accordingly. I'm guessing labeled by folder name.

What kind of server side troubles could I run into?

I'm aware that there may be many issues. Even general ideas would be could so I can then research further. Thanks!

Also, I would be programming in Ruby on Rails but I think this question applies accross any language.

Answer 1

For those many images it has to be a serious app.. thus giving you the liberty to suggest a piece of software running on the client (something like yahoo mail/picassa does) that will take care of 'managing' (network interruptions/resume support etc) the upload of images.

For the server side, you could process these one at a time (assuming your client is sending them that way)..thus keeping it simple.

Answer 2

take a peek at http://gallery.menalto.com
they have a dozen of methods for uploading pictures into galleries. You can choose ones which suits you.

Answer 3

Either have a client app, or some Ajax code that sends the images one by one, preventing timeouts. Alternatively if this is not available to the public. FTP still works...

Answer 4

I'd suggest a client application (maybe written in AIR or Titanium) or telling your users what FTP is.

deviantArt.com for example offers FTP as an upload method for paying subscribers and it works really well.

Flickr instead has it's own app for this. The "Flickr Uploadr".

Answer 5

There's no reason why you couldn't handle this kind of thing with a web application. There's a couple of excellent components that would be useful for this:

1. Uploadify (based on jquery/flash)
2. plupload (from moxiecode, the tinymce people)

The reason they're useful is that in the first instance, it uses a flash component to handle uploads, so you can select groups of files from the file browser window (assuming no one is going to individually select thousands of images..!), and with plupload, drag and drop is supported too along with more platforms.

Once you've got your interface working, the server side stuff just needs to be able to handle individual uploads, associating them with some kind of user account, and from there it should be pretty straightforward.

With regards to server side issues, that's really a big question, depending on how many people will be using the application at the same time, size of images, any processing that takes place after. Remember, the files are kept in a temporary location while the script is processing them, and either deleted upon completion or copied to a final storage location by your script, so space/memory overheads/timeouts could be an issue.

If the images are massive in size, say raw or tif, then this kind of thing could still work with chunked uploads, but implementing some kind of FTP upload might be easier. Its a bit of a vague question, but should be plenty here to get you going ;)