The .NET WebClient class is nice and all, but it doesn't parse the JavaScript in the page.
What I mean is, sometimes a page is forwarded to a different page via JavaScript, or sometimes DOM elements get added to the page via JavaScript with an onload event. I need to deal with both of these situations.
What can I do to deal with this? I'm using WPF if that matters.
To do this in managed code would essentially mean writing your own browser. I expect that to do what you want the simplest approach would be to drop a WebBrowser control into the page (essentially re-hosting IE) and use that. There are other hostable controls, of course - but that is the simplest for Windows.
You can use the WebBrowser control. This approach is not as "lightweight" as using the WebClient, but short of writing your own javascript parser this is probably the easiest solution.