How to transfer binary data through multiple http server?

Question

Well, the question is not intended to be that big. Let me explain the scenario: I have two http servers. server A is accessible to end user by web browser, while server B is internal server which can only be accessed by server A. If server B generate some big jpeg image in local disk, and server A has no way to access server B's filesystem directly, how to let end user see those image without firstly storing those image data in server A temporarily?

I run PHP on server A and perl on server B, but this should not matter. I need a general pattern for implementing this.

Answer 1

obviously we can't just delivery those path to image to server A and eventually to end user.

I think that's the only way to go, but you don't need to physically save the files on server A. In PHP: If server A can talk to server B on filesystem level (i.e. through a network share), server A could fetch the data from server B and pass it through to the user:

header("Content-type: image/jpeg"); // Make sure you send the right headers
$file = fopen("/path/to/server/b/huge/image.jpg", "r");
fpassthru($file);  // or deliver chunks using fread()
fclose($file);

If there is only an internal http connection, you would change the 2nd line to something like

$file = fopen("http://serverb.local/huge/image.jpg", "r");

if this method is too slow for you or not convenient to set up, you would have to use (S)FTP, SCP or something similar. FTP is available in PHP natively; the other protocols are probably easiest to simply call from the PHP script using exec().

depending on your scenario and use frequency, you may want to employ some sort of caching on Server A, so that this operation doesn't have to be repeated every time.

If your servers are hosted in a datacenter, make sure that traffic between them is free or not too expensive.

This is the leanest way I can think of to let the user "see" an image without it clogging Server A.

Answer 2

There's a number of solutions, depending on how much control you have and just how isolated server B is from server A.

One way is for server B to generate the images on a network volume shared with server A. Server A would have read-only access to provide a bit of security. Then server A can directly access the files in question. The advantages over writing a pass-through program on server A is its likely to be faster, as the only extra overhead is the network drive, and it allows server B to remain completely isolated from server A.

This does assume that server A does not need to request that server B generate the images, that they're just there.