Fundamentally, WCF messages are XML, since the S.S.C.Message class uses the Xml InfoSet as the base message representation.
So yes, in some way you need to "wrap" your binary content into an XML envelope (which doesn't need to be a SOAP envelope, mind you, depending on how your binding is configured).
That said, note that this doesn't preclude streaming to deal with large message payloads without buffering the entire message in memory; WCF still allows you to do this, though sometimes it's not greatly obvious how it works.
Since you're working with a custom channel, you've got one of two choices, I think:
- Have your own channel add the XML wrapper around your binary content before passing it upwards or
- Create a custom MessageEncoder that adds it in for you automatically.
In either case, if you're dealing with large messages, you'll want to make sure you use the MessageEncoder overloads that handle Streams instead of buffers, as they are the ones that give you the option of doing coordinated streaming with the service implementation.