Multiple certificates for HTTPS on a software NLB'd IIS7 cluster

Question

We're currently trying to set up a HTTPS with multiple certificates. We've had some limited success but we're getting some results I can't make any sense of...

Basically we have two servers on our NLB (10.0.51.51 and 10.0.51.52) and two IPs assigned to our NLB (10.0.51.2 and 10.0.51.4) and we have IIS listening on both of these IPs with a different wildcard certificates (To avoid giving out public IP's let's say A:443 routes to 10.0.51.2:443 and B:443 routes to 10.0.51.4:443). We also have a Cisco router using port address translation to route port 443 from two external IP's to these internal NLB IPs.

The weird thing is, this works if we request A:443 or B:443, but if you go internally on 10.0.51.51:443, 10.0.51.52:443, 10.0.51.2:443 or 10.0.51.4:443 you ALWAYS get the same SSL cert. This cert was in the past assigned to *:443 but we've made sure there's no * bindings anymore defined in IIS.

When i run "netsh http show sslcert" after trimming out all the irrelevant stuff I get:

IP:port                 : 0.0.0.0:443 
Certificate Hash        : <Removed: Cert 1>

IP:port                 : 10.0.51.2:446 
Certificate Hash        : <Removed: Cert 3 - Another site>

IP:port                 : 10.0.51.3:446 
Certificate Hash        : <Removed: Cert 3 - Another site>

IP:port                 : 10.0.51.4:443 
Certificate Hash        : <Removed: Cert 2>

Which tells me that the * binding is still in there, which is a bit weird, but I can't see why that would prevent the other from working (Or even more more strangely why the request through the router would work).

It's got me wondering whether it's actually treating the requests as the machine's IP rather than the NLB IP, but unfortunately our dev environment is only a single server which sorta reduces the amount of trial/error I can take to this (Since all I can test on is a live environment) without convincing management to buy more servers for the test environment - which is something I'm trying.

Does anyone have any idea:

1. Why there's a difference between internal and through the router?
2. Why the internal request is getting the wrong cert?
3. How I can remedy this so that we get the same behavior on both sides?

Answer 1

I ended up tracking the problem down. Leaving this as a hint for anyone else who falls in the same trap...

The problem was caused by us using a shared configuration model on our IIS servers. When setting up a HTTPS binding this appears to only actually bind it on the box you're managing it on (Leaving the other completely unbound). Since our * binding still existed it was catching it on the server we didn't do through the UI and just let pick up the shared config.

Crazy bad luck with single-affinity NLB sent us down the garden path after the router being the cause by making our internal requests go to one server and our external requests to another.

We ended up finding this by running "netsh http show sslcert > certs.txt" on both servers and diff'ing the outputs.

Going forwards our plan is to no longer use the IIS UI for SSL configuration instead following the steps below:

1. Install the certificates on each server.
2. Run a command-line binding of the SSL port "netsh http add sslcert ipport=?:? certhash=? appid=?" (ip:port is easy to work out, certhash can be copied from the "certificate hash" section of the server certificates page, appid can be copied from an existing IIS binding on the netsh http add sslcert)
3. Edit the IIS ApplicationHost.config file directly to add the bindings without the UI being involved.

Our understanding is this will prevent a repeat of this error.