Using CURL within a loop to download a file, 1st one works, 2nd one times out

Question

Morning all,

I am using CURL to download an image file within a loop. The first time it runs fine and I see the image appear in the directory.

The second time it fails with a timeout, despite it being a valid URL.

Can anyone suggest why it always fails on the 2nd time and how to fix it?

The snippet of code is:

// download image

$extension = "gif";
$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 90);
curl_setopt($ch, CURLOPT_URL, $imgurl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
echo $imgurl . " attempting to open URL ";
$i = curl_exec($ch);
if ( $i==false ) {
  echo curl_errno($ch).' '.curl_error($ch);
}
$image_name=time().'.'.$extension;
$f = fopen('/fulldirectorypath/' . $image_name ,'w+');
fwrite($f,$i);
fclose($f);

I have put an echo in there to display the $IMGURL, to check it is valid, and upped the timeout to 90 secs, but it still fails. This is what I see on screen:

http://images.eu-xmedia.de/thumbnails/34555861/5676051/pt=pic,lang=2,origfile=yes/image.gif attempting to open URL 28 Operation timed out after 90 seconds with 0 bytes received

an empty file is created in my directory.

thanks alot,

Greg

Answer 1

If you are hitting the same site over and over, they may have something which times out requests coming in too fast. If you are looping, try adding a brief sleep after each iteration.

One thing you should probably look in to using is multi-curl. It runs Curl operations in parallel. The section on Curl Functions on php.net documents them. I'd suggest looking at this tutorial. Someone out there has probably written an object-oriented abstraction of curl and multi-curl.

Answer 2

Try adding curl_close($ch); after fclose($f);. It seems likely to me that it is trying to reuse the connection, which is in return causing the issue.