How to save webpage to string with cookies support (httpWebRequest)

Question

I need to read webpage and store its content in string for further processing.
Sounds simply but I have problem with cookies support.

Opened page says I need browser supporting cookies (or turned on).
I've made method trying do that via httpWebRequest - which normally works to me but I've come to a standstill with those unfortunate cookies...

Any idea how to make it working?

Here is my method:

string ReadHtml (string address, string encoding) {
   Uri url = new Uri(address);

   CookieContainer cookieContainer  = new CookieContainer();
            HttpWebRequest httpWebRequest  =     (HttpWebRequest)HttpWebRequest.Create(url);
   httpWebRequest.AllowAutoRedirect = true;
            httpWebRequest.KeepAlive   = true;
            httpWebRequest.CookieContainer  = cookieContainer;
            httpWebRequest.UserAgent   = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
            httpWebRequest.Method    = "GET";
            HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();

   // Code Page
   Encoding enc = Encoding.GetEncoding(encoding);  

   // Read content
   StreamReader loResponseStream = new     StreamReader(webResponse.GetResponseStream(),enc);
   string   lcHtml    = loResponseStream.ReadToEnd();

   webResponse.Close();
   loResponseStream.Close();

   return lcHtml;

  }

Answer 1

I'm not sure I quite understand the question... are you talking about taking the contents of a webpage and storing them into a cookie? Although I wouldn't exactly recommend this, if you need to, use:

HttpCookie cookie = new HttpCookie("cookieName");
cookie.Value = ReadHtml("http://www.google.com/", "utf-8");
cookie.Expires = DateTime.Now.AddHours(1);
Response.Cookies.Add(cookie);

Josh

Answer 2

The page (until you r making request) sends you a cookie and at another request this page is trying to read thatcookie. When you don't provide cookie, then page think that your browser desn't support that mechanism.

I recommend you to:

1. Open Fiddler application
2. Check what cookies are saved (trying to save)

The solution is to provide back to page exacly that cookie, that this page sends you.