web scraping in txt mode

Question

Hi All,

I am currently using watir to do a web scraping of a website hiding all data from the usual HTML source. If I am not wrong, they are using XML and those AJAX technology to hide it. Firefox can see it but it is displayed via "DOM Source of selection".

Everything works fine but now I am looking for an equivalent tool as watir but everything need to be done without a browser. Everything need to be done in txt file.

In fact right now, watir is using my browser to emulate the page and return me the whole html code I am looking. I would like to the same but without the browser.

Is it possible ?

Thanks Regards Tak

Answer 1

Your best guess would be to use something like webscarab and capture the URLS of the AJAX requests your browser is doing.
That way, you can just grab the "important" data yourself by simulating those calls with any HTTP library