PHP Curl - Get data generated with AJAX

Question

Hi, I want to get data generated by an AJAX request. In this page http://www.fipe.org.br/web/index.asp?p=51&aspx=/web/indices/veiculos/default.aspx there are some html selects. When the user click on the first one (Marca), the second one is filled. I want to get this data.

This is my code:

<?php
$curl = curl_init();
$postData = array('ddlAnoValor' =>  0,
                                    'ddlMarca' => 1,
                                    'ddlModelo' => 0,
                                    'ddlTabelaReferencia' => 123,
                                    'txtCodFipe' => '');
$result = null;
$httpResponse = null;

curl_setopt($curl, CURLOPT_URL, 'http://www.fipe.org.br/web/indices/veiculos/default.aspx?p=51');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_REFERER, 'http://www.fipe.org.br/web/indices/veiculos/introducao.aspx');
curl_setopt($curl, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, $postData);

$result = curl_exec($curl);

$httpResponse = curl_getinfo($curl, CURLINFO_HTTP_CODE);
if($httpResponse == '404') {
    throw new exception('This page doesn\'t exists.');
}

echo $result;

curl_close($curl);
?>

Page request header

Host: www.fipe.org.br

User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.13) Gecko/20100916 Iceweasel/3.5.13 (like Firefox/3.5.13)

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

X-MicrosoftAjax: Delta=true

Cache-Control: no-cache, no-cache

Content-Type: application/x-www-form-urlencoded; charset=utf-8

Referer: http://www.fipe.org.br/web/indices/veiculos/default.aspx?p=51

Content-Length: 9415

Cookie: __utma=106123796.1351303072.1287075522.1287075522.1287075522.1; __utmb=106123796; __utmc=106123796; __utmz=106123796.1287075522.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); ASPSESSIONIDAADQDQRD=EKBEJHEDKCIOAAHNFFMLGMKO

Pragma: no-cache

But I always get the form as result. I've tried to set cookie but cookies.txt file is always empty. I don't know if this cookie is required. cookies.txt has 777 permission. What am I doing wrong? Thank you.

Answer 1

If you look at the post variables (use the net panel on firebug to do this) when using the form on the site, you will see that it contains some variables which you are not submitting with your PHP code, such as _VIEWSTATE and _EVENTVALIDATION.

I guess that these relate to the session established by the browser when displaying the form, and I further guess that if these and their related variables are not present then the server will return the full page HTML including the form.

You could try to simulate these variables, but I suspect you are doomed to fail.

Ideally you should contact the site and ask them how you can retrieve the information you are looking for. Perhaps they have a webservice which exposes it?