Following a Javascript link when scraping from a remote site using PHP

Question

Given remote page:

http://example.com/paged_list.aspx

which uses a Javascript function call to display several pages of tabular data:

javascript: show_page(1) javascript: show_page(2)

and so on. Users click on the page links to display each page, which triggers a reload but with no query string, ie the URI remains the same.

In scraping this site, it would be useful to have a way to obtain subsequent pages but there is no obvious way to specify a page number in the request (passed to file_get_contents()).

Is there any way to:

1. Open a remote web address.
2. Call a known javascript function at that address.
3. Return the results?

Answer 1

Emulating JS in PHP would be the tough route. Much easier to analyze the JS source and determine the URL target of the background AJAX operation. Should then be a fairly easy task to pull the entire data set into your PHP script by calling the URL and modifying args as needed.

Answer 2

Your best bet would be to try to reverse engineer the javascript function/ajax calls to their php script in order to do your own request to their server with the correct arguments.

You can use many firefox addons to make your life easier at reversing all of that (e.g. firebug by looking at network activity)