views:

235

answers:

1

My apologies, I've actually asked this question multiple times, but never quite understood the answers.

Here is my current code:

while($resultSet = mysql_fetch_array($SQL)){            
$ch = curl_init($resultSet['url'] . $fullcurl); //load the urls and send GET data
            curl_setopt($ch, CURLOPT_TIMEOUT, 2);           //Only load it for two seconds (Long enough to send the data)
            curl_exec($ch);                                 //Execute the cURL
            curl_close($ch);                                //Close it off 
} //end while loop

What I'm doing here, is taking URLs from a MySQL Database ($resultSet['url']), appending some extra variables to it, just some GET data ($fullcurl), and simply requesting the pages. This starts the script running on those pages, and that's all that this script needs to do, is start those scripts. It doesn't need to return any output. Just the load the page long enough for the script to start.

However, currently it's loading each URL (currently 11) one at a time. I need to load all of them simultaneously. I understand I need to use curl_multi_*, but I haven't the slightest idea on how cURL functions work, so I don't know how to change my code to use curl_multi_* in a while loop.

So my questions are:

How can I change this code to load all of the URLs simultaneously? Please explain it and not just give me code. I want to know what each individual function does exactly. Will curl_multi_exec even work in a while loop, since the while loop is just sending each row one at a time?

And of course, any references, guides, tutorials about cURL functions would be nice, as well. Preferably not so much from php.net, as while it does a good job of giving me the syntax, its just a little dry and not so good with the explanations.

EDIT: Okay zaf, here is my current code as of now:

        $mh = curl_multi_init(); //set up a cURL multiple execution handle

$SQL = mysql_query("SELECT url FROM urls") or die(mysql_error()); //Query the shell table
                    while($resultSet = mysql_fetch_array($SQL)){   

        $ch = curl_init($resultSet['url'] . $fullcurl); //load the urls and send GET data
        curl_setopt($ch, CURLOPT_TIMEOUT, 2);           //Only load it for two seconds (Long enough to send the data)
        curl_multi_add_handle($mh, $ch);
    } //No more shells, close the while loop

        curl_multi_exec($mh);                           //Execute the multi execution
        curl_multi_close($mh);                          //Close it when it's finished.
+1  A: 

In your while loop, you need to do the following for each URL:

  • create a curl resource by using curl_init()
  • set options for resource by curl_setopt(..)

Then you need to create a multiple curl handle by using curl_multi_init() and adding all the previous individual curl resources by using curl_multi_add_handle(...)

Then finally you can do curl_multi_exec(...).

A good example can be found here: http://us.php.net/manual/en/function.curl-multi-exec.php

zaf
Oh I don't know why I didn't think of that. How would I be able to add a multiple curl handle from the handles in the while loop?
Rob
Interesting. Would I be able to do this:`$mh = curl_multi_init();while getting rows from db {curl_multi_add_handle($mh,$resultSet['url']);}curl_multi_exec($mh);`
Rob
Yes, try it, it should work. If not then post back and we'll look at it.
zaf
Thanks, will do. Give me a couple of minutes
Rob
Oh another question, will I need to remove the handles before closing the multi_exec?
Rob
Generally good practice to remove the resources but you probably get away with automatic 'garbage collection' when the script ends. But don't tell anyone I told you that.
zaf
Haha, okay thanks. Let me try it.
Rob
Alright zaf, check my current code. Would that work?
Rob
Hmm, I'm worried that the references to "$ch" may get lost. Try it and if it works, great. Otherwise it may be wiser to have 2 loops. The first adding curl resources to an array for example and another loop that adds the resources from array to the multi curl handle. This is less error prone and will be easier for you to maintain in the future.
zaf
Yeah, thats what I was worried about, as well. I'll try it first, if it doesn't work I'll set up an array, which will be nice for closing the handles. Then again, I may set up the array anyway, just so I CAN close the handles with ease. Thanks for the help zaf.
Rob
Extra note: Just tested, doesn't work. I'll have to set up an array. Perhaps can you help me with that, as well?
Rob
Sure. if you call the array "$ch" then just add the curl resources with "$ch[]=curl_init($resultSet['url'] . $fullcurl);". In the second just use a "foreach($ch as $curl){..." to loop thru and add to the curl_multi.. And if you want to remove the curl resources just do another foreach at the end to remove them.
zaf
Yeah I don't have much experience with arrays, looking it up on php.net right now. I'll comment again when i figure out how to define the array :)
Rob
Hm, zaf, looking at it again, I can't quite understand what to do there. Can you explain more thoroughly? Sorry.
Rob
Actually zaf, got it now. Thanks for all the help
Rob
+1 for working it out.
zaf