After spending a few weeks on this problem, I have pretty much a definitive answer:
Can you replace XML/SOAP with a custom binary serialization for WCF?
The short answer: no
The long answer: yes, but you have to rewrite almost all of the transport layer interfaces so you might as well just create a custom IHttpHandler and avoid WCF all together.
Microsoft, in attempting to follow the SOAP standard (and the standard is responsible for this problem in my opinion) breaks one of the simple programming rules -> separation of responsibilities in the layers. WCF/SOAP may appear to be a elegantly layered protocol and transport, but the reality is, is that there are intricate connections between the layers. This means that WCF is an extremely complex communication protocol that provides transport, security, reliability, publishing, serialization and other features that are all interdependent at some level. If all you want is a transport channel, WCF/SOAP adds a tremendous amount of complexity.
I'm sure I could start huge rants about SOAP yes/no, but I finally realized that WCF/SOAP is not what I needed for my application.
-Jeff