Getting HTML from a page behind a login

Question

This question is a follow up to my previous question about getting the HTML from an ASPX page. I decided to try using the webclient object, but the problem is that I get the login page's HTML because login is required. I tried "logging in" using the webclient object:

WebClient ww = new WebClient();

 ww.DownloadString("Login.aspx?UserName=&Password=");

 string html = ww.DownloadString("Internal.aspx");

But I still get the login page all the time. I know that the username info is not stored in a cookie. I must be doing something wrong or leaving out an important part. Does anyone know what it could be?

Answer 1

Well does opening the page in a brower with "Login.aspx?UserName=&Password=" normaly work?
Some pages may not allow login using data provided in the url, and that it must be entered in the login form on the page and then submitted.

Answer 2

@Fire Lancer: I asked myself that same question during my tests, so I checked, and it does work from a browser.

Answer 3

Try setting the credentials property of the WebClient object

WebClient ww = new WebClient();
ww.Credentials = CredentialCache.DefaultCredentials;
ww.DownloadString("Login.aspx?UserName=&Password=");
string html = ww.DownloadString("Internal.aspx");

Answer 4

The only other reason I can think of then is that the web page is intentionally blocking it from loggin in. If you have access to the code, take a look at the loggin system used to see if theres anything designed to block such logins.

Answer 5

As the aspx page I was trying to get was in my own projct, I could use the Server.Execute method. More details in my answer to my original question

Answer 6

Just pass valid login parameters to a given URI. Should help you out.

If you don't have login information you shouldn't be trying to circumvent it.

public static string HttpPost( string URI, string Parameters )
      {
         System.Net.WebRequest req = System.Net.WebRequest.Create( URI );
         req.ContentType = "application/x-www-form-urlencoded";
         req.Method = "POST";
         byte[] bytes = System.Text.Encoding.ASCII.GetBytes( Parameters );
         req.ContentLength = bytes.Length;
         System.IO.Stream os = req.GetRequestStream();
         os.Write( bytes, 0, bytes.Length );
         os.Close();
         System.Net.WebResponse resp = req.GetResponse();
         if ( resp == null ) return null;
         System.IO.StreamReader sr = new System.IO.StreamReader( resp.GetResponseStream() );
         return sr.ReadToEnd().Trim();
      }

Answer 7

Use Firefox with the LiveHttpHeaders plugin.
This will allow you to login via an actual browser and see EXACTLY what is being sent to the server. My first question would be to verify that it isn't expecting a POST from the form. The example URL you are loading is sending the info via a querystring GET.

Answer 8

Use Fiddler to see the HTTP requests and responses that happen when you do it manually through the browser.