PHP Curl: read incrementally

Question

I have a PHP script that's supposed to take in a URL and search a database for a cached version of the URL. If it finds it, it prints out the cached version to the user, otherwise it downloads it using cURL and echos it to the user. Currently, the downloading script looks something like this:

// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1); 
// grab URL and pass it to the browser
$data = curl_exec($ch);
$mimetype = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
// close cURL resource, and free up system resources
curl_close($ch);

mysql_query("INSERT INTO `cache_files` (`key`, `original_url`, `file_data`, `mime_type`) VALUES (".
 "'" . mysql_real_escape_string($key) . "', " .
 "'" . mysql_real_escape_string($url) . "', " .
 "'" . mysql_real_escape_string($data) . "', " .
 "'" . mysql_real_escape_string($mimetype) . "')"
);

if (isset($_GET["no_output"])) die;

header ("Content-Type: " . $mimetype);
header ('Expires: '.gmdate('D, d M Y H:i:s \G\M\T', time() + 157680000), true);
header ('Last-Modified: '.gmdate('D, d M Y H:i:s \G\M\T'), true);
header ('Cache-Control: public; max-age=157680000');
header ('Pragma: ');
print $data;

This is currently working fine, however, large files are not send to the end user until they are 100% downloaded, which causes incremental rendering not to be triggered. I want to know if there's a way that cURL can pass the data to the user as it downloads, but also have the data available in a string for script consumption.

Answer 1

you could use the CURLOPT_WRITEFUNCTION callback to hook into cURL's response data handling. example (via http://www.php.net/manual/en/function.curl-setopt.php#26239):

<?php

function myPoorProgressFunc($ch, $str) {
  global $fd;

  $len = fwrite($fd, $str);
  print('#');
  return $len;
}

curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'myPoorProgressFunc');

instead of writing the data to a file and outputting '#', you could write the data to the browser and save it in string. however, i would be cautious with latter for large files.

Answer 2

Here's a crude and incomplete (but working) class to test the use of the CURLOPT_WRITEFUNCTION option. According to the libcurl documentation, the function given for this option "gets called by libcurl as soon as there is data received that needs to be saved." So the content received by curl should be passed to the server as it is received. When it is actually displayed would be dependent, of course, on the server and the browser.

$url = 'http://stackoverflow.com/';
$curl_test = new CurlCallbackTest($url);
file_put_contents('some.file', $curl_test->content);

class CurlCallbackTest
{
    public $content = '';

    public function __construct($url)
    {
        $ch = curl_init($url);
        curl_setopt($ch, CURLOPT_WRITEFUNCTION, array($this, 'showContent'));
        curl_setopt($ch, CURLOPT_HEADER, false);
        curl_exec($ch);
        curl_close($ch);
    }

    private function showContent($ch, $data)
    {
        $this->setContent($data);
        echo htmlspecialchars($data);
        return strlen($data);
    }

    private function setContent($data)
    {
        $this->content .= $data;
    }
}