How to retrieve an html redirected webpage programmatically ?

Question

Hello,

I have done this code to login,to retrieve and show a webpage :

  // login info array
        string postData = "user_name=tler";
        postData += "&user_password=lodvader";
        byte[] data = Encoding.ASCII.GetBytes(postData);

        // web request
        WebRequest req = WebRequest.Create("http://www.lol.com/login.php");
        req.Method = "POST";
        req.ContentType = "application/x-www-form-urlencoded";
        req.ContentLength = data.Length;

        // stream response to string
        Stream newStream = req.GetRequestStream();
        newStream.Write(data, 0, data.Length);
        newStream.Close();
        StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream(), Encoding.GetEncoding("iso-8859-1"));

        string responseString = reader.ReadToEnd();

        // retrieve text within title
        Regex rx = new Regex(@"(?<=<title>).+?(?=</title>)");

        var variable = rx.Matches(responseString);

        // output
        Console.WriteLine(variable[0]);

        Console.ReadLine();

But, the following page after login is an html redirect like :

<meta http-equiv="refresh" content="3; URL="bb.php">

How to follow this link and retrieve next page ?

Answer 1

Just send a new WebRequest to the bb.php file. Make sure that you use the same CookieContainer since I presume that login.php uses cookie-based sessions to remember you. Check out the HttpWebRequest.CookieContainer property. This requires you to cast your WebRequest to a HttpWebRequest.

Added: (Couldn't write example code in the comment.)

I'm just making code up without proofing now...

var cookies = new CookieContainer(); 

var firstReq = (HttpWebRequest)WebRequest.Create(".../login.php");
firstReq.CookieContainer = cookies;

var secondReq = (HttpWebRequest)WebRequest.Create(".../bb.php");
secondReq.CookieContainer = cookies

Answer 2

There is a property of HttpWebRequest called AllowAutoRedirects. Set this to true. Also there is a property called MaximumAutomaticRedirections. Set that to some allowable value to make sure all of them are followed.

Answer 3

You can't do it a easy way, since the meta tag is read by the client and executed.

In this case, when you're using HttpWebRequest, the request doesn't care about the functions the text may have.

So you need to do another request to the page in the URL attribute (bb.php).

-

If the server did the redirection you wouldn't have the problem.

Answer 4

Hello,

I have found the time to finish it, here the response ( i tried to be as clear as possible ) :

        // Cookie for our session
        var cookieContainer = new CookieContainer();

        // Encode post variables
        ASCIIEncoding encoding=new ASCIIEncoding();
        byte[] loginDataBytes = encoding.GetBytes("user_name=belaz&user_password=123");

        // Prepare our login HttpWebRequest
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://blabla.fr/verify.php");
        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        request.CookieContainer = cookieContainer;
        request.ContentLength = loginDataBytes.Length;

        // Write encoded post variable to the stream
        Stream newStream = request.GetRequestStream();
        newStream.Write(loginDataBytes, 0, loginDataBytes.Length);
        newStream.Close();

        // Retrieve HttpWebResponse
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        // Link the response cookie to the domain
        cookieContainer.Add(new Uri("http://blabla.fr/"),response.Cookies);

        // Prepare our navigate HttpWebRequest, and set his cookie.
        HttpWebRequest requestProfile = (HttpWebRequest)WebRequest.Create("http://blabla.fr/bb.php");
        requestProfile.CookieContainer = cookieContainer;

        // Retrieve HttpWebResponse
        HttpWebResponse responseProfile = (HttpWebResponse)requestProfile.GetResponse();

        // Retrieve stream response and read it to end
        Stream st = responseProfile.GetResponseStream();
        StreamReader sr = new StreamReader(st);
        string buffer = sr.ReadToEnd();