Easiest way to get web page source code from pages that require logins -- C#

Question

So I play an online game that's web based and I'd like to automate certain things with it using C#. Problem is that I can't simply use WebClient.DownloadData() because I need to be logged in to actually recieve the source. The other alternative was to use the built-in web browser control but that doesn't give me access to source code. Any suggestions?

Answer 1

You can set the login credentials on the webclient using its Credentials property before calling DownloadData:

WebClient client = new WebClient();
client.Credentials = new NetworkCredential("username", "password");

EDIT: As mjmarsh points out, this will only work for sites that use a challenge-response method of authentication as part of a single request (I'm so used to dealing with this at work, I hadn't considered the other types!). If the site uses forms authentication (or indeed any other form of authentication), this method will not work as the authentication is not part of a single request - multiple requests are needed that you will need to handle yourself.

Answer 2

I don't think NetworkCredentials will work in all cases. This only works with "Basic" or "Negotiate" authentication.

I've done this before with an internal website for some load testing, but sounds like you are trying to "game" the game. For that reason I won't go into details but the login to the site is probably being done in the form of an HTTP POST when you hit the login button.

You'd have to trap the POST request and replicate it in your code and make sure that your implementation maintains the session state as well, because if the game site is written well at all it will make sure that the current session has logged in before doing anything game related.

Answer 3

Network credentials will not work as mjmarsh has already pointed out.

While web scrapping we come across lot of pages where login is needed. One of the approaches I use is install fiddler and monitor the POST and GET packets while manually logging in the site. This allows you to find out how the browser emulates the login. Then you need to recreate the same process by Code.

For example, most web servers use cookies to assume the session is authenticated. So you can use the credentials to post UserName and Password on the web site and record the Cookie. This cookie can then be used to access any further details on the web site.

Please check following link to check out more about Advanced Web Scrapping:

http://krishnan.co.in/blog/post/Web-Scraping-Yahoo-Mail.aspx

In this blog, you will find how to authenticate into Yahoo account and then read the page after authentication.