Building a website to handle lots of file transferring and storage

Question

I'm planning to build a website that will involve transferring large amounts of different types of files (documents, video, pictures, audio, etc) and storing them on the backend.

1) While starting out with a small user base and minimal cash, what's an good, economic way to handle this type of website? With the very little I know right now, I would just start by looking into different hosting companies and the pricing for their different packages. Is this a good idea or am I heading in entirely the wrong direction?

2) What type of web development technologies/languages immediately comes to mind when thinking about transferring and storing these kinds of files?

Thanks so much in advance for all your help!

Answer 1

I would checkout services like Amazon ec2 it offers incredibly cheap file storage, I recently started using it for my servers off site backups and have saved alot of money.

Answer 2

First the obvious ones. Make sure you clearly understand your document storage and bandwidth needs and be careful about what happens when you go over your allocation with your hosting provider. These things can get VERY expensive very fast.

Second, just about any modern language will handle this sort of application. I would start with the environment you are most comfortable with.

Don't forget about backups (and the bandwidth that will take).

One last piece of advice. DON'T STORE THE DOCUMENTS IN A RELATIONAL DATABASE. Sorry for shouting, but it's important. Store the documents on a file system and store pointers to them in the database.

Answer 3

Amazon S3 is a great Service/API for file storage, but a simple asset server could be knocked up to store and retrieve files very quickly.

It's a bit early maybe for production use but couchdb has an attachments API that might do most of the heavy lifting for you.... though if it really is JUST files your serving, then the filesystem and a decent static webserver will win hands down.

Answer 4

One additional thing you might want to consider (for redundancy only -- not production access) is storing all the files in a database.

Should you move servers, add servers, etc, the process would be simplified by a script that would access the database and detatch the files to disk, or upload them to Amazon EC3, etc.

This may not be viable for large files, but I have had success doing it on a video site that contains many small clips. Roll-out is a breeze and I never have to worry about having the latest uploaded videos, etc.