tags:

views:

58

answers:

2

Hi, I'm trying to use Watin to parse google search results. However watin is unable to find elements i the Google Search Result page. When I view the source it's because the page is generated off javascript so the search results are not sent over the wire in html.

However when I open up Firebug (in Firefox) I am able to parse the html that gets generated by the javascript.

Does anyone know how I can get Watin to do the same so I'm able to parse the results?

Thanks :)

A: 

Could you use the google search API instead?

http://code.google.com/apis/ajaxsearch/documentation/#fonje

Saxon Druce
I think you have to pay for searches.
I think it is free, although you have to comply with their terms (http://code.google.com/apis/ajaxsearch/signup.html). If your use does not comply with their API terms, then scraping content using watin probably doesn't comply with the general Google terms either (eg section 5.3 of http://www.google.com/accounts/TOS).
Saxon Druce
I need to use Watin for this solution. Thanks for the input though :)
A: 

It may be a matter of timing. If javascript is generating the data, you may be checking for your data before it is written. Try running it in debug, stepping through and waiting until you know the items exist in the source before using WatiN to test for it.

I would also suggest posting code so we can see exactly what you're trying to do.

Daaron