HttpWebRequest SSL only working on some websites

Question

Hello,

I'm doing some experimenting with HttpWebRequest, and need to get it working with SSL. It does work on some websites (PayPal, for instance), but not the ones I actually want it working on, such as this community website (URL is also in the code sample). Why is this? The certificates (I'm assuming this is where the problem might be) look awful similiar.

    public static bool AcceptAllCertificatePolicy(object sender,
                                            X509Certificate certificate,
                                            X509Chain chain,
                                            SslPolicyErrors sslPolicyErrors)
    {
        return true;
    }

    static void LoginTest()
    {

        ServicePointManager.ServerCertificateValidationCallback += AcceptAllCertificatePolicy;

        HttpWebRequest req = (HttpWebRequest)
        WebRequest.Create("https://steamcommunity.com");

        req.Method = "GET";

        req.CookieContainer = new CookieContainer();

        HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

        Console.WriteLine("Cookies:" + resp.Cookies.Count + "\r\n" + resp.ResponseUri);
        Console.WriteLine(resp.Headers);

        resp.Close();
    }

Thanks

EDIT: The URL was incorrect, sorry. Problem persists though.

EDIT2: I forgot to mention that while it doesn't give me any errors, it doesn't return any cookies either. That's what I'm really after. www.paypal.com gives me 7 cookies, response headers being

Pragma: no-cache
Cache-Control: private
Date: Sat, 31 Oct 2009 12:10:29 GMT
Expires: Thu, 05 Jan 1995 22:00:00 GMT
Set-Cookie: LANG=sv_SE%3bSE; expires=Tue, 29-Oct-2019 12:10:30 GMT; domain=.paypal.com; path=/,cookie_check=yes; expires=Tue, 29-Oct-2019 12:10:30 GMT; domain=.paypal.com; path=/,navcmd=_home-general; domain=.paypal.com; path=/,consumer_display=USER_HOMEPAGE%3d0%26USER_TARGETPAGE%3d0%26USER_FILTER_CHOICE%3d7%26BALANCE_MODULE_STATE%3d1%26GIFT_BALANCE_MODULE_STATE%3d1%26LAST_SELECTED_ALIAS_ID%3d0; expires=Sun, 31-Oct-2010 12:10:30 GMT; domain=.paypal.com; path=/,cwrClyrK4LoCV1fydGbAxiNL6iG=_Yi2Io2xQuP5-kmjAhh1KUJVMH6VTFfnf1Zlw1ONf41yAAUyLXUiLZ_9GJcPZmmNMl6kgpaG14eNnVA6jYlkyaK7h2IPqoPUsUobSg-gzEf4NsHrbwWzuCe8W50EIPW4ABaAkG%7c3EWGI8_elkol97q63exWWZMOA02gaDHo4_Y_7lKC_xHE2kWR8DZGXAmqs4uv075KRh6OW0%7ctJQpXroB_JJPd9EvK7Xx17DfiF9lz5vs06ThWWluWxwM3-WO82KChgIGdC080wwm5Q6vL0%7c1256991030; domain=.paypal.com; path=/,navlns=0.0; expires=Fri, 26-Oct-2029 12:10:30 GMT; domain=.paypal.com; path=/,Apache=10.190.11.252.1256991029926042; path=/; expires=Fri, 18-Sep-03 05:42:13 GMT
Server: Apache
Vary: Accept-Encoding
Strict-Transport-Security: max-age=500
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

while steamcommunity.com results in 0 cookies being set,

Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Cache-Control: no-cache
Content-Type: text/html; charset=UTF-8
Date: Sat, 31 Oct 2009 12:14:25 GMT
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Server: Apache

even though it should set at least one.

Answer 1

The problem is that the certificate is not valid for https://www.steamcommunity.com but for https://steamcommunity.com. So all you have to do is change your url:

WebRequest.Create("https://steamcommunity.com");

Look at the Issued to header. For paypal it is Issued to: www.paypal.com, while for your site it is Issued to: steamcommunity.com

---

UPDATE:

new WebClient().DownloadString("https://steamcommunity.com"); // works
new WebClient().DownloadString("https://www.steamcommunity.com"); // throws invalid certificate

Answer 2

Install firebug for firefox. Clear your cookie cache, and then navigate to the site. Check in firebug to see what is the exact request sent by the browser. Then replicate the exact same request using code. This means replicating all headers sent by the browser.

Some servers will not send cookies if they dont think that it is a browser issuing the request. So, putting all the request headers should make it appear to the site as if a browser is requesting the pages.