pratical challenge: retrieving cookies from this site in .Net

Question

Hi folks, I have a pratical question here. I have to access to a site and get the cookies information to reuse it in subsequent navigation. I don't have any browser control on my side, because everything is supposed to run on a server. With some site the task is pretty easy, but there are some site that send back cookies in a way I cannot manage to intercept.

On example is Italian highways site: I can't understand how they sent back cookies and most important, how can I capture those cookies from a .Net application.

I've tried both WebClient object, looking into header, in this way:

var wc = new WebClient();
var htmlMainPage = wc.DownloadString(new Uri(AutostradeMainSite));
string cookies = wc.ResponseHeaders["Set-Cookie"];

but I don't get any result. Even by looking into header, there's no cookies there.

Then I tried with HttpWebRequest object, but I wasn't able to retrieve the cookies:

HttpWebRequest wr = (HttpWebRequest)HttpWebRequest.Create(AutostradeMainSite);
wr.Method = "GET";
HttpWebResponse response = (HttpWebResponse)wr.GetResponse();
var cookies = response.Cookies

What am I doing wrong?

Analyzing the main page of the site with some developer tools for IE or Chrome, I can see that some cookies are sent to the browser, but I cannot see them neither in the header, nor in the javascript... How can it work? Thanks in advance for any help.

Answer 1

The WebClient class can't handle cookies out of the box, although it's pretty easy to create a derived version that supports cookies. You can do it with the HttpWebRequest class, but you need to set the CookieContainer property:

HttpWebRequest req = WebRequest.Create("http://www.google.com") as HttpWebRequest;
req.CookieContainer = new CookieContainer();
HttpWebResponse res = req.GetResponse() as HttpWebResponse;
CookieCollection cookies = res.Cookies;

Answer 2

The site may be detecting that you're using an unknown "browser", and assuming you cannot accept cookies, so it doesn't even provide one. The answer would be to craft a custom WebRequest that looks like it comes from IE 8 instead of the .NET WebClient. You should be able to do this by setting a value for the User-Agent key of WebClient.Headers.