Best way to grab website content externally

Question

There's a search site whose search results are generated dynamically by javascript. So the user enters a query, and the site displays the content on the page, without refreshing.

I need to grab those search results programmatically (say from a Java program or a perl/python script).

So ideally, I can launch my program with 100 queries as user inputs, and then the program would hit that website with each query and spit out on my screen all the search results as returned by the website.

The obvious problem is that the site is in javasript instead of simple HTML, so sending a URL request and parsing the resultant output is not going to work (as the source code for this page is always just a bunch of reference to various .js files).

Given the above conditions, what are my options?

Answer 1

Install Firebug, study the requests that are made by the site's javascript and mimic them in your program. Chances are there is a single request that needs to be made and the resulds would come in some nice form like JSON

Answer 2

Unless the search provider gives you an API to work with (either via backchannel agreement or a publicly available one), then nothing you do will be likely to work for very long.

You may go to great pains to fool the site into believing you are an ordinary website user. Then, they will make some minor change to how their site works (because they have no idea someone is using it in the fashion you are) and all of a sudden your hack won't work. Sometime later, they may notice that you are using them in this fashion, and detect your usage and flat out block it.

Basically, unless they give you an API, you are essentially stealing, and should expect to receive all the courtesy that deserves... none.

Lest you think I am judging you, I'll let you know I speak from experience ;)

Answer 3

Javascript does http requests almost just like a browser does, once you figure out what they are you can try to re-create them in perl/python/etc. With Firefox+Firebug you can see the requests in the 'Net' panel.

Things you might have to take into account are user-agent string, cookies, the fact that sometimes the returned data is meant to be run/interpreted by Javascript etc. Maybe your language of choice has a nice httpbrowser class you can use?

---

Just took a look, searching for IBM, took the post data from Firebug, replaced newlines with '&' and put it after the request url:

[http://bcode.bloomberg.com/sym/dwr/call/plaincall/searchMgr.search.dwr?callCount=1&windowName=&c0-scriptName=searchMgr&c0-methodName=search&c0-id=0&c0-e1=string:ibm&c0-e2=string:&c0-e3=number:100&c0-e4=number:0&c0-e5=boolean:false&c0-param0=Object_SearchCriteria:{search:reference:c0-e1,%20filter:reference:c0-e2,%20limit:reference:c0-e3&,%20start:reference:c0-e4,%20allSources:reference:c0-e5}&batchId=4&page=%2Fsym%2F&httpSessionId=&scriptSessionId=FBC68693A4E1BC08D6E0DDFBDF6D0860]

but it returns

throw 'allowScriptTagRemoting is false.';
//#DWR-REPLY
if (window.dwr) dwr.engine.remote.handleBatchException({ name:'java.lang.SecurityException', message:'GET Disallowed' });
else if (window.parent.dwr) window.parent.dwr.engine.remote.handleBatchException({ name:'java.lang.SecurityException', message:'GET Disallowed' });

and no data.. So it looks like you have to script a post request. Looking at their restrictions and guidelines, maybe you should just get in touch and ask if there's a public API?