I am looking to achieve the fastest and scalable communication between several servers. I've developed a custom TCP socket server for this purpose with (IOCP) as I think it would be the fastest communication in .NET, but I don't know if it crashes on high load, or if my solution is the fastest communication channel for .NET applications. Is there any open source solution out there to meet my needs?
What's the fastest, scalable, reliable communication channel in .NET Applications?
I would expect that a custom TCP or UDP server running a custom protocol that is ideally tailored for your particular needs would be the most efficient solution but since you're not really telling us much about WHAT you need to communicate it's rather difficult to answer.
Of course the solution also depends on what technologies you're comfortable with; writing scalable TCP servers is hard and you need to profile your solution at all stages of development with a realistic number of test clients and simulated data flow, etc. Writing custom reliable or semi reliable UDP protocols is even harder
It may well be that a 'less fast' solution will do and will be easier to implement as someone else will have done all of the hard work for you. Again the only way to know is to profile something that resembles a realistic representation of your problem.
There are many, many things that impact the performance of a network application, beginning with architectural design decisions, protocol choice, network congestion, down to message serialization, memory allocations, etc. There is no "one size fits all" way of network communication that makes an application automatically scalable, robust under high load, etc.
Lower level protocols have less overhead, that is right. But higher level protocols may be much better for some use cases, because they already provide some features which you otherwise would have to implement yourself. And, as I said, the protocol is not the only thing that affects the performance.
You need to gather your requirement, identify the bottlenecks, design an appropriate architecture, choose a protocol at the right abstraction level, and profile, profile, profile.