How to download web pages in C# with the same behavior as using a web browser?

Question

I have this site I want to get information from. To get the file that I need, I need to go through several pages with cookies and to submit a form.

I already have a code that deals with the cookie, but to submit the form is much trickier since the site is in ASP.NET, which means the form contains the view state.

Is there some smarter way to simulate surfing the web? Maybe some kind of web browser API, or some kind of service that lets you browse from page to page with cookies and form values?

Thanks,

Answer 1

Use WebRequest and CookieContainer classes.

Answer 2

Use WatiN. It's a way to automate browsers (really for unit testing) so you get all of the cookie handling, JavaScript execution, etc. of a real browser because it is a real browser.

See this question: Best Method for Website Automation