How to mass-fetch pages from a website behind authentication in PHP

Question

Yes, yes, i know user and password.

I need some trick in php to get logged into a website and retrieve some images/contents, like a normal website.

Obviously with a curl o file_get_contents it doesn't work because i'm not authenticated.

How i can do?

The authentication is normal HTTP auth with POST.

Edit: ok thanks for help it works!

I post working code here for future reference

//login and set cookie
$curl = curl_init();
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_COOKIEFILE, "cookiefile");
curl_setopt($curl, CURLOPT_COOKIEJAR, "cookiefile"); # SAME cookiefile
curl_setopt($curl, CURLOPT_URL, "url in which there is the login form"); 
curl_setopt($curl, CURLOPT_POSTFIELDS, "user=test&password=test&someparam=somevalue"); //put here the post/get values
$output = curl_exec($curl);

echo $output;

//finally fetch my content
curl_setopt($curl, CURLOPT_URL, $url_to_fetch); 
$output = curl_exec($curl);
echo $output;

curl_close ($curl);

Answer 1

Use the browser to authenticate yourself, export the cookies and use them via curl. Until the session lasts you should impersonate your user.

I'm in a hurry and can't provide you code just now, but I think this directions can help you

You can use the CURLOPT_COOKIEFILE option to specify the file in which you stored the cookies.

As stated in the php manual:

The name of the file containing the cookie data. 
The cookie file can be in Netscape format, or just 
plain HTTP-style headers dumped into a file. 

Answer 2

You can authenticate with curl. Curl allows for sending POST variables to login, and also basic HTTP authentication.