How can I read in C# XML from another page while logged in to the site

Question

I have logged in to the site with my webbrowser and whenever I try to call

WebClient myWebClient = new WebClient();
string str = myWebClient.DownloadString("http://localhost/myxml.aspx");
Response.Write(str.ToString());

Or

XmlTextReader reader = new XmlTextReader(url);
while (reader.Read()) {
 Response.Write(reader.ReadOuterXml());
}

Response.Write returns me the login page.

Is there away to attach user SessionId to WebClient or XmlTextReader or how can I request another page in C# with current logged user?

Answer 1

You'll need to use an object that can handle storing cookies. In this case, you'll need the HttpWebRequest class. You'll also need a CookieContainer to manage authentication cookies.

To do this you would:

1. Create a CookieContainer object (a cookie jar) that you can keep track of throughout the scope of every request you make.
2. Create an HttpWebRequest that logs into the site you're accessing.
3. Use the CookieContainer you created in step 1 for every subsequent request.

Below is an example of how to use the HttpWebRequest, HttpWebResponse, and CookieContainer classes together to make a simple request that will set some cookies, and then using those cookies on a subsequent request. The rest should be easy assuming everything is well formed markup ;)

CookieContainer cookieJar = new CookieContainer();

var webRequest = (HttpWebRequest)HttpWebRequest.Create("http://www.google.com");
webRequest.CookieContainer = cookieJar;

var webResponse = webRequest.GetResponse();

using (var reader = new StreamReader(webResponse.GetResponseStream()))
{
    Response.Write(reader.ReadToEnd());
}

var anotherWebRequest = (HttpWebRequest)HttpWebRequest.Create("http://www.google.com/search?q=stackoverflow.com");
anotherWebRequest.CookieContainer = cookieJar;

webResponse = anotherWebRequest.GetResponse();

Another option (if you really want to use the WebClient class) would be to parse out the ResponseHeaders property of the class once you've made your request and include the appropriate cookies in your next request. This is a little more involved though since it requires you to manage your cookies manually.

Since I'm assuming that you want to be able to traverse your web responses as XML, I suggest you look into the open source library, HtmlAgilityPack. It allows you to send in markup from a web site that is (most likely) not well-formed, or has some sort of invalid markup in it, and then fixes the invalid parts so that you can traverse it like XML.

Answer 2

While doing some screen scraping, I had the same issue. I was requesting a Classic ASP app on an IIS server (I could tell by some of the headers that the server reponded with). The way I supported an ongoing session was by enabling Cookies on the WebClient. Theres no toggle for it, you have to subclass WebClient to get it to work.

    public class CookieAwareWebClient : WebClient
    {
        protected CookieContainer _container = new CookieContainer();

        public CookieContainer Cookies
        {
            get { return _container; }
            set { _container = value; }
        }

        protected override WebRequest GetWebRequest(Uri address)
        {
            HttpWebRequest httpRequest = base.GetWebRequest(address) as HttpWebRequest;      

            if (httpRequest.CookieContainer != null)
            {
                if (httpRequest != null)
                {
                    CookieCollection newCookies = 
                        GetUniqueCookies(
                                address
                                ,httpRequest.CookieContainer.GetCookies(address)
                                );
                    foreach (Cookie c in newCookies)
                        httpRequest.CookieContainer.Add(c);
                }
            }
            else
                httpRequest.CookieContainer = this.Cookies;

            return (WebRequest)httpRequest;
        }

Note: this isn't a unique solution, I found this out there on the web myself, but I've implemented the solution and it works really well.