How do you minimize the number of threads used in a tcp server application?

Question

I am looking for any strategies people use when implementing server applications that service client TCP (or UDP) requests: design patterns, implementation techniques, best practices, etc.

Let's assume for the purposes of this question that the requests are relatively long-lived (several minutes) and that the traffic is time sensitive, so no delays are acceptable in responding to messages. Also, we are both servicing requests from clients and making our own connections to other servers.

My platform is .NET, but since the underlying technology is the same regardless of platform, I'm interested to see answers for any language.

Answer 1

The modern approach is to make use of the operating system to multiplex many network sockets for you, freeing your application to only processing active connections with traffic.

Whenever you open a socket it's associated it with a selector. You use a single thread to poll that selector. Whenever data arrives, the selector will indicate the socket which is active, you hand off that operation to a child thread and continue polling.

This way you only need a thread for each concurrent operation. Sockets which are open but idle will not tie up a thread.

• Using the select() and poll() methods
• Building Highly Scalable Servers with Java NIO

Answer 2

G'day,

I'd start by looking at the metaphor you want to use for your thread framework.

Maybe "leader follower" where a thread is listening for incoming requests and when a new request comes in it does the work and the next thread in the pool starts listening for incoming requests.

Or thread pool where the same thread is always listening for incoming requests and then passing the requests over to the next available thread in the thread pool.

You might like to visit the Reactor section of the Ace Components to get some ideas.

HTH.

cheers, Rob

Answer 3

A more sophosticated aproach would be to use IO Completion ports. (Windows) With IO Completion ports you leave to the operating system to manage polling, which lets it potentially use very high level of optimization with NIC driver support. Basically, you have a queue of network operations which is OS managed, and provide a callback function which is called when the operation completes. A bit like (Hard-drive) DMA but for network.

Len Holgate wrote an eccelent series on IO completion ports a few years ago on Codeproject: http://www.codeproject.com/KB/IP/jbsocketserver2.aspx

And I found an article on IO completion ports for .net (haven't read it though) http://www.codeproject.com/KB/cs/managediocp.aspx

I would also say that it is easy to use completion ports compared to try and write a scaleable alternative. The problem is that they are only available on NT (2000, XP, Vista)

Answer 4

If you were using C++ and the Win32 directly then I'd suggest that you read up about overlapped I/O and I/O Completion ports. I have a free C++, IOCP, client/server framework with complete source code, see here for more details.

Since you're using .Net you should be looking at using the asynchronous socket methods so that you don't need have a thread for every connection; there are several links from this blog posting of mine that may be useful starting points: http://www.lenholgate.com/archives/000440.html (some of the best links are in the comments to the original posting!)