How to make a static URL point to different servers?

Question

Im building multi-server support for a file upload site Im running. When images are uploaded.... they are thumbnailed and stored on the main front-end server, until cron executes (every 10 mins) and moves them to storage servers, so for the first 10 mins, they will reside, and be served off the main front-end server.

When a file is uploaded, users are given embed codes... which is a thumbnail url + link to the full size, which is a html page. So it might be something like http://www.domain.com/temp_content/filename.jpg which links to http://www.domain.com/file-ID

Except in 10 mins, http://www.domain.com/temp_content/filename.jpg wont exist, it will be http://server1.domain.com/thumbs/filename.jpg

if the user grabbed the original code... the thumb file will be broken.

I COULD move the file to its destination, without cron, but that will take time, and will lag the script until the move is complete. I also dont like to have users running commands like that, I'd rather have the server do them at regular intervals.

Anything else I can?

Answer 1

You could use a mod_rewrite command in your .htaccess to check if the file in temp_content exists, and if it doesn't, have it redirect them to the new location.

Answer 2

Have you considered a database storing the image_name/image_location, and a generic PHP script to serve the images from the database-details?

Answer 3

Really, given your situation the only option I can see is to give your users the actual URL's, since I'm sure you will be able to know them. You will then need to notify the users that they cannot actually use the link for 10 minutes.

In an idea world, I would see you putting this file directly to its final resting place, given your need to allow users access to the link.

Answer 4

How about some JavaScript?

<img src="http://www.domain.com/temp_content/filename.jpg"
onerror="this.src='http://server1.domain.com/thumbs/filename.jpg'"&gt;

Answer 5

I don't see how this will be an issue. If you're having the user upload files their locations must be stored in a record somewhere correct? And your pages must be checking these records when you generate them right? Why not just add another field that specifies what server they're on. Then just make sure the cron task updates the record once it's done moving the file?

Answer 6

This is the kind of thing that begs for a network hack. Basically, a custom routing thing that is pointed to by the domain and forwards packets to the appropriate host. It's ugly

In all honesty though, this is a great question for ServerFault, because they're the folks who would know how to set up the topology to enable this kind of thing.

Answer 7

Use a database to map image names to locations. Modern databases do caching. IF you find that performance is bad, you could also have a simple hash-table cache in memory - with this you could store could store ~2 million name->location mappings in 500MB of RAM (assuming ~256 bytes per mapping).

For the script that serves the files, it could

1. Actually serve the file (read bytes from the location and send the bytes out to the client).
2. Redirect the client to actual location on a different server.

Answer 8

I currently move all uploaded media on our website to amazon s3/cloudfront after about 10 minutes time, and do a combination of 2 things to redirect users to the new location.

for public assets (thumbs, etc...), we cache the parent item's definition with the new location of media (eg. server1.site.com/media/1.jpg).

for private assets, requests are made to a script that checks authentication, then issues a 302 redirect to the auth'd s3 url.

long story short, store the new location in memcache, have mod_rewrite pipe to a script on 404 for the original file, then 302 redirect to the new location.

Answer 9

Cron does not "decide" anything about the load balancing of the files across servers. Cron executes a command at an appointed time.

Where is the logic that actually does the load balancing of images across servers? Can the load balancing portion of that logic be done by the script that handles the upload? The cron job can still handle the actual moving of the image from the upload server to the final server.

This way the user could be presented with a "temp" location to verify the upload. After confirming the upload, the user would get a "final" link to the image, and a message about "may need to wait 10 minutes for the image to be 'live' at that location."

Answer 10

very simple:

i don't like your paths for your files, so i changed it. ^_^

create the link to go to the main storage immediately. http://www.domain.com/file/filename.jpg

on the main server use a ruleset like so

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} /file/(.+)
RewriteRule ^/file/.+ /temp_content/%1 [L,R]