For one you'll have issues with different activation, threading models, buffers, framing, security and serialisation at the least, in integration or hand-rolled code. It is typical of any UDP/TCP or config/attribute work in any language or environment.. And it will still need tweaking according to server or client needs, simple the nature of vast possibilities in I/O work.
Completely agree WCF is way too bloated and much slower in comparison to .NET Remoting (which is slower than managed socket code which is again much slower than native iocp code) to hand rolled code.. But it is extra work and if you managed the above in a general generics library (not more powerful templates which makes it harder in .NET, ie. you have to design types in odd ways + it is imperative to use interface) than you've done well over say a man-year with some sophisticated plumbing a la message-queuing (it is isn't impossible but imho and retrospect never worth doing 'generically'..).
Alternatives are MSMQ, something MS is increasingly betting more of its software on, or half-assed open source bits that just never satisfy the model you require completely. Tibcos, ActiveMQs, you name it. Even going lower level, the best of all is widely considered to be C++ boost::asio, and although considered top, like all out things out there it isn't appropriate for everything. And it is a pretty remarkable design so I'd watch out for a potential time-wasting exercise..
On .NET I'd first adopt whatever 50-thread service MS is pushing 'today', and if not do the 'yet again similar code' and look for similar concepts to equip your toolset down the line when it reappears (like every new I/O lib or framework does every 5 years or so).