Is there a way to determine if a browser supports NTLM without having NTLM enabled for the particular site or directory in IIS and without showing a login dialog/pop-up? Preferably, determine this using ColdFusion or perhaps some combination of JS and CF. I'd prefer not to restrict this to just IE as other browsers (such as FF) support NTLM authentication.
If you request a page and the page returns 401 and says it only accepts NTLM, and then the browser sends another HTTP packet trying to respond to it, then it supports NTLM. You don't have to make IIS do this -- you could have any page where you can set the response codes and headers request NTLM. If you don't get another request, it means that the client couldn't authenticate this way.
You could detect this on the client by putting this request in an IFrame, then in the outer page checking to see what happened in the iframe.
Building on Lou's answer, you could make a cfhttp request within a try/catch block. You then check the response headers to determine your next steps.
Unfortunately, you may be forced to use browser sniffing and a white-list.