webclient response is different to the IE response

Question

When i call download data on my webclient, the response is different to when i open it in IE. I am trying to call www.yell.com searching for a type of company in a certain location. When i get my response it finds 2 companies but doesnt actually display them, it just says 2 results found. In IE it actually displays the results. The URLs are exactly the same so i dont know what could be causing this?

Answer 1

• Maybe the website uses JavaScript to load additional data after the main page has loaded? Get Fiddler and check what passes through the pipes.

• Is there some sort of CSS/Javascript hide/unhide work done on page load? Check with IE8's built-in script debugger.

• Maybe there's a CSS stylesheet loaded which hides those results, or are they physically missing from the response?

• How do the webclient/IE headers differ? Some sites send different content depending e.g. on User-Agent header ("If this is not IE/FF/Opera/Webkit, it's probably a bot, send different content"). Pay attention to other header fields as well. Again, check using Fiddler.

• Are the results in IE displayed as HTML? Is it not some Flash/Java/Silverlight/whatever object? (strange but possible)

• Is there some cookie action going on? Some sites set a cookie on access, and use it to keep state - e.g. "you haven't gone through the search page, no results for you". Again, Fiddler to the rescue.

• Does the site work OK in other browsers? If not, what's broken? (not a problem per se, but can give you a hint where a problem could be)