non-blocking requests with php and curl not working with sleep

Question

Hey dear stackoverflow members,

I have set up a small script to use the known cUrl feature of curl_multi_* code to provide asynchronous non-blocking requests , this is a rough version of the code:

$mch = curl_multi_init();
$ch = curl_init();

url_setopt($ch, CURLOPT_URL,      $url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);       

curl_multi_add_handle($mch ,$ch);
$running = null;
do {
    curl_multi_exec($mch ,$running);
} while($running > 0);

curl_multi_remove_handle($mch,$ch);
curl_close($ch);

basically this is used to send one request as a parallel request to allow the code to keep running while another script is called via cURL , I do not care for the result therefore I used no code to handle it I just want the other script to be called while the code after the curl call will keep running.

first of all I don't understand why do I need the while loop , wouldn't calling multi_exec once for one request work ? since not using the loop does not run the script on the $url(I checked).

secondly and my biggest problem , this should be asynchronous , I tried putting sleep(10) on the other script and the calling script waits 10 seconds before continuing , I dont get it this should be non-blocking and should keep running.

what am I doing wrong?

Answer 1

Lets start from the end.

When your PHP finishes running your script, it performs a cleanup and kills off all the still running curl handles. It is important to remember that.

Now, each curl request is divided into multiple steps, like initializing the connection, connecting, writing data, reading data, closing the connection. The middle steps may be repeated. And I'm sure that actually there are more complex things happening behind the scenes, but overall this description should be correct.

curl_multi_exec gives control to each of the handles in the multi handle and allows it to perform its next step, whatever it may be. And then the function returns.

This explains why you are not seeing the request without the loop. It happens because your handle has not yet reached the step where it is able perform the actual connection. Your PHP script finishes running, performs the cleanup, which in turn kills the handle, that got no chance to do anything at all.

So, now its clear that you need to give your handle chance to do something. Running curl_multi_exec over and over again is one way to achieve it.

This explains why you are experiencing the wait, these multi handles are not really asynchronous, they just look like they are. Your cycle gives the handle chance to run and the cycle will keep running until the request has finished (which in your example takes 10 seconds).

Short summary: you need to find another solution to your problem (which is to keep running other code, while curl is executing) :)

Answer 2

This one works good for me:

function get_web_page( $url )
{
$options = array(
    CURLOPT_RETURNTRANSFER => true,     // return web page
    CURLOPT_HEADER         => false,    // don't return headers
    CURLOPT_FOLLOWLOCATION => true,     // follow redirects
    CURLOPT_ENCODING       => "",       // handle all encodings
    CURLOPT_USERAGENT      => "spider", // who am i
    CURLOPT_AUTOREFERER    => true,     // set referer on redirect
    CURLOPT_CONNECTTIMEOUT => 120,      // timeout on connect
    CURLOPT_TIMEOUT        => 120,      // timeout on response
    CURLOPT_MAXREDIRS      => 10,       // stop after 10 redirects
);

$ch      = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err     = curl_errno( $ch );
$errmsg  = curl_error( $ch );
$header  = curl_getinfo( $ch );
curl_close( $ch );

$header['errno']   = $err;
$header['errmsg']  = $errmsg;
$header['content'] = $content;
return $header;
}

As a function you can call it over and over. (as asyncronous threads) So really all you would need to do is write a while loop to handle receiving and handling all the received curls.. once that is done your script can finish, avoiding the halt at end of code issue.

Another point.. you dont need to format your while loops like this..

do { ... } while (conditional)

you can just use:

while (conditional) { ... }