Is threre a way to get cookies without reading entire page?

Question

I am trying to post some data and get some data back from a website. The site does not expose any webservices or API's and I have no way of modifying the site. The only option is HTML screen scraping. In order to post the data, I first need to log in, get a cookie go to another page get another cookie and go to the final page to post the data. I don't need any html from the first two pages but need the cookies in order to do the final post. I am using a System.Net.HttpWebRequest to navigate the site. Is it possible to receive just the cookies (Server headers) without downloading the entire page? Right now I am using GetReponse() method to get the response stream.

Answer 1

No, not if you're sending a POST request. You can retrieve just the headers by sending a HEAD request, but then you won't be able to post the required login credentials.

Answer 2

I think it depends on the site. In theory if you use HEAD request, the site should respond with the headers but with no data seems to be what you need, In reality it all depends on the actual web site

Answer 3

You should (in theory) be able to set the Method property on your HttpWebRequest to HEAD and only get the headers back.

Answer 4

with the win32 api you can read just the response header and drop the connection after. Look into wininet.

As for the HEAD predicate, it might work in some situations while in others it wont. Some pages actually check for the predicate and take action based on that.