Hi guys
I'm trying to scrape a quote engine but HttpWebResponse returns only the "please wait" screen, not the final result.
Is there a way to make the request wait for the response? Status codes won't work because the "please wait" screen returns 200. - I guess it redirects after it retrieved the results from the database but that redirect comes after the initial 200.
AllowAutoRedirect is true but it doesn't make any difference in this case.
Has any of you seen this before?
My guess is the site can tell you are scraping and is telling you to bug off.
The site is probably using either Javascript or a meta refresh tag to load the ultimate destination page. Neither of those are directly supported by HttpWebRequest. You've got the source to the "please wait" screen - read it to see exactly how they are redirecting your browser to the end result. There may be a URL you can parse out of the content for the "please wait" screen. Once you have that URL, you can just load it.