Retrieve a CSV file generated from a table by javascript

Question

First, let me say that I am not familiar with the terminology so, if you see something, by all means, help me improve the wording.

What I want to do is retrieve a CSV file that is generated by a website, apparently based on a table.

The site in question has two drop boxes from which one can select the queries and then, based on `onchange=', a search is made and a table is filled.

With the table filled, a button appears, which can then be pressed and the CSV file, containing the fields, is offered to download.

After poking around with the page, I was able to find and construct the URL responsible to retrieve the CSV file. It is something like:

http://www.example.com/exportCSV.action?field1=3&field2=5

The problem is, if I try to `curl' it, a empty CSV file is retrieved, with just the headers. So, I think that the actual content must be inside the table which is filled using the normal web interface.

The last call from the javascript function that generates the CSV is:

window.open("exportCSV.action?"+fields)

Is there a way to satisfy the initial search so, when I try to curl the `CSV url' I can get a filled CSV, and not a empty one?

Answer 1

This rather sounds like that web site is not accepting your cURL request. Some try to limit their services to “real” browsers only.

Try using a debugging tool like FireBug to have a look at the actual data that the JavaScript sends and receives over the network.

I assume you are doing your cURL call right? Passing parameters, especially on the command line, can be a bit tricky. Make sure you escape the URL correctly, for example with single quotes:

curl 'http://www.example.com/exportCSV.action?field1=3&field2=5'

Else, the & character and possibly the question mark as well might get interpreted by your shell.