curl problem, can't download full web page

Question

With this code I'm trying to download this web page: http://www.kayak.com/s/...

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,'http://www.kayak.com/s/search/air?ai=kayaksample&do=y&ft=ow&ns=n&cb=e&pa=1&l1=ZAG&t1=a&df=dmy&d1=4/10/2010&depart_flex=exact&r1=y&l2=LON&t2=a&d2=11/10/2010&return_flex&r2=y');

 curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 curl_setopt($ch, CURLOPT_FAILONERROR, true);
 curl_setopt($ch, CURLOPT_REFERER,"http://wwww.google.com"); 

 $content = curl_exec ($ch);
 echo $content;

You can see the demo at: http://www.pointout.org/test.php
As you can see the part with prices is missing.
What could be wrong?

Answer 1

This is not going to work the way you think it will. The reason is the prices are not in the initial HTML response that you get. Rather, there is some Javascript magic occurring which is using AJAX to load the prices when the page is loaded.

Answer 2

Your page content is no simple html, therefore it need to be rendered by a browser before you can download it. So instead of curl, use a browser for the download. You can use iMacros for Firefox and write a short iMacros macro (with SAVEAS TYPE=CPL to save the complete website...) and trigger it via the command line.