i am working on a website which deploy on web farms to serve high traffic. where should i store user uploaded files? is it wise to store uploaded files in the file system of the same website and synchronize these files in all web servers(web farm)? or should i use another server to store all uploaded files in this server to store files in a central location? if separate file server will be a better choice, than how can i pass files from web server to that file server efficiently? or should i upload files directly to that file server?
views:
160answers:
3
A:
For security reasons you want to serve the files from a different sub-domain anyways. This is easier by dedicating one server for these files, if the traffic for these files warrants it, but that also creates a single point of failure and bottleneck which you probably want to avoid.
You can have all web servers serve the files from a central file server or local cached copies. This depends a lot on if the files ever change.
Sam
2010-03-12 16:52:21
how facebook stores user uploaded images? my case is a bit same with facebook photos.i think facebook also uses separate file server because all the images in user albums are serve from different domain rather than from facebook.comwhat u recommend if one have to provide facebook photos like functionality?
Inam Jameel
2010-03-12 17:19:22
A subdomain can be served from the same phyiscal box or set of boxes. Also facebook has a huge user base, so if you're talking about a scale like that then you want to store on Amazon S3 and serve via CloudFront. If you can't go that far, then make sure you serve the files from multiple web servers and either use a caching mechanism which will cache-upon-request or push the files to all web servers so there is no single bottleneck or point of failure (cache-on-demand has a single point of failure for non-cached files, but less of a bottleneck).
Sam
2010-03-12 17:34:08
got it samone last questionis there any tool available which automatically synchronizes uploaded files to all the servers in web farm.my web servers will be windows 2008 servers and my website is ASP.NET MVC site.i cant go for amazon s3 and for a while cant use any mechanism for caching files.thanks for your feedback.
Inam Jameel
2010-03-12 17:39:21
@Inam Jameel, I'm sure there are lots of options to auto-sync files. You'll be best off asking a new, specific, question about this.
Sam
2010-03-12 19:41:56
A:
Have you considered using Amazon S3 to store and serve user-uploaded files?
Otherwise, a central file server in your data center makes sense. You could mount the storage locally on each of your web servers using NFS or similar.
However you choose to store these files, you should always serve user-provided files from a different domain to avoid cross-domain exploits.
Matthew
2010-03-12 17:02:15
nope currently there is no plan to use Amazon s3 or something similar. we have already 3 dedicated servers connected via gigabit LAN somewhere.how facebook stores user uploaded images? my case is a bit same with facebook photos.i think facebook also uses separate file server because all the images in user albums are serve from different domain rather than from facebook.comwhat u recommend if one have to provide facebook photos like functionality?
Inam Jameel
2010-03-12 17:19:49
A:
You asked about how Facebook stores photos. You might want to read the following article:
Needle in a haystack: efficient storage of billions of photos
Joneph O.
2010-03-12 17:40:50