Thread pool sizing for network-bound tasks

Question

What are the basic rules for sizing a thread pool which contains tasks that:

• are always network-bound;
• access external services with different throughput and latency.

My main concern is using the bandwith optimally ( i.e. don't process tasks serially, but don't open 1200 network connections either );

Answer 1

Is your network access synchronous or asynchronous?

If your network access and request processing is asynchronous, then the thread pool size can be => Number of available cores + 1.

UPDATE: By available cores, I mean the number of physical processors available on the system. A thread pool is required even for a server using async I/O to take advantage of multiple physical processors.

If your network access and request processing is synchronous, then there is no hard-and-fast rule for sizing the thread pool. It is always better to make the thread pool size configurable in this case. A guestimate for a default value, given that your request processing is not CPU bound, could be:

(Request Processing Latency/Network Latency) * (Number of Available cores + 1) with a maximum value of 4 * Number of Avaliable cores.

Answer 2

I'm not an expert but shouldn't you have one thread per task? The threads will block/poll until data is available. If you're waiting on a connection then start a thread to handle each connection in the connection handler.