PHP - Downloading very large files with fsockopen(), fgets() and feof()

Question

I have a simple download function in a class that might be dealing with files of many hundreds of MB at a time from an Amazon Web Services bucket. The whole file cannot be loaded into memory at once, so it must be streamed directly to a file pointer. This is my understanding as this is the first time I've dealt with this issue and I'm picking things up as I go along.

I've ended up with this, based on a 4k file buffer which simple testing showed was a good size:

        $fs = fsockopen($host, 80, $errno, $errstr, 30);

        if (!$fs) {
          $this->writeDebugInfo("FAILED ", $errstr . '(' . $errno . ')');
        } else {
          $out = "GET $file HTTP/1.1\r\n";
          $out .= "Host: $host\r\n";
          $out .= "Connection: Close\r\n\r\n";
          fwrite($fs, $out);

          $fm = fopen ($temp_file_name, "w");
          stream_set_timeout($fs, 30);

          while(!feof($fs) && ($debug = fgets($fs)) != "\r\n" ); // ignore headers

          while(!feof($fs)) {
            $contents = fgets($fs, 4096); 
            fwrite($fm, $contents);
            $info = stream_get_meta_data($fs);
            if ($info['timed_out']) {
              break;
            }
          }
          fclose($fm);
          fclose($fs);

          if ($info['timed_out']) {
            // Delete temp file if fails
            unlink($temp_file_name);
            $this->writeDebugInfo("FAILED - Connection timed out: ", $temp_file_name);
          } else {
            // Move temp file if succeeds
            $media_file_name = str_replace('temp/', 'media/', $temp_file_name);
            rename($temp_file_name, $media_file_name);
            $this->writeDebugInfo("SUCCESS: ", $media_file_name);
          }
        }

In testing it's fine. However I have got into a conversation with someone who is saying that I am not understanding how fgets() and feof() work together, and he's mentioning chunked encoding as a more efficient method.

Is the code generally ok, or am I missing something vital here? What is the benefit that chunked encoding will give me?

Thanks for any advice,

Mark...

Answer 1

Your solution seems fine to me, however I have a few comments.

1) Don't create a HTTP packet yourself, i.e. don't send the HTTP request. Instead use something like CURL. This is more fool proof and will support a wider range of responses the server might reply with. Additionally CURL can be setup to write directly to a file, saving you doing it yourself.

2) Using fgets may be a problem if you are reading binary data. Fgets reads to the end of a line, and with binary data this may corrupt your download. Instead I suggest fread($fs, 4096); which will handle both text and binary data.

2) Chunked encoding is a way for a webserver to send you the response in multiple chunks. I don't think this is very useful to you, however, a better encoding that the webserver might support is the gzip encoding. This would allow the webserver to compress the response on the fly. If you use a library like CURL, it will tell the server it supports gzip, and then automatically decompress it for you.

I hope this helps

Answer 2

Dont deal with sockets, optimize your code an use curl library. like this:

$url = 'http://'.$host.'/'.$file;
// create a new cURL resource
$fh = fopen ($temp_file_name, "w");
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FILE, $fh); 
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
fclose($fh);

Answer 3

And the final result in case it helps anyone else. I also wrapped the whole thing in a retry loop to decrease the risk of a completely failed download, but it does increase the use of resources:

      do {
        $fs = fopen('http://' . $host . $file, "rb");

        if (!$fs) {
          $this->writeDebugInfo("FAILED ", $errstr . '(' . $errno . ')');
        } else {
          $fm = fopen ($temp_file_name, "w");
          stream_set_timeout($fs, 30);

          while(!feof($fs)) {
            $contents = fread($fs, 4096); // Buffered download
            fwrite($fm, $contents);
            $info = stream_get_meta_data($fs);
            if ($info['timed_out']) {
              break;
            }
          }
          fclose($fm);
          fclose($fs);

          if ($info['timed_out']) {
            // Delete temp file if fails
            unlink($temp_file_name);
            $this->writeDebugInfo("FAILED on attempt " . $download_attempt . " - Connection timed out: ", $temp_file_name);
            $download_attempt++;
            if ($download_attempt < 5) {
              $this->writeDebugInfo("RETRYING: ", $temp_file_name);
            }
          } else {
            // Move temp file if succeeds
            $media_file_name = str_replace('temp/', 'media/', $temp_file_name);
            rename($temp_file_name, $media_file_name);
            $this->newDownload = true;
            $this->writeDebugInfo("SUCCESS: ", $media_file_name);
          }
        }
      } while ($download_attempt < 5 && $info['timed_out']);