keeping 1000 tcp connections open inspite of very few(10/20) actual communications.

Question

For a socket server, what are the costs involved in keeping 1000s of tcp connections open, but only few clients are actually communicating? I'm using single threaded poll/select based server. Also, what is the max open-socket limit of linux kernel(plz don't give conf file suggestions. i want theoretical limit)?

Answer 1

Costs: the socket fd; the kernel data structures corresponding to it; the TCP connection tuple (protocol;source:port;destination:port) and its data structures; the socket send and receive buffers; and the data structures in your code that keep track of connections.

The limit depends on your configuration but it's generally much higher than 1000.

Answer 2

The main cost (for typical TCP/IP stack implementations) is in transmit and receive buffers living in kernel memory -- the recommended size in this good (if maybe dated) short essay is socket buffer size = 2 * bandwidth * delay. So, if the delay is 100 ms (ping time, a round trip of two delays, would then be 200 ms), and the bandwidth is about a gigabit/second (call it 100 MB per second for ease of computation;-), to get top performance you need a socket buffer size of about 20 MB per socket (I've heard of no kernel configures so generously by default, but you can control the buffer size per-socket just before you open the socket...).

A typical socket buffer size might be, say, 256 KB; with the same hypothetical 100 ms delay, that should be fine for a bandwidth of up to 5 megabit/second -- i.e., with that buffer size and delay, you're not going to be fully exploiting the bandwidth of even a moderate quality broadband connection ("long narrow pipes" -- large bandwidth and large delay, so their product is quite large -- are notoriously difficult to exploit fully). Anyway, with that buffer size, you'd be eating up 256 MB of kernel memory by keeping 1000 TCP sockets open and connected (albeit inactive).

I don't think (at least on a 64-bit kernel) there are other intrinsic limits besides the amount of RAM allocated to the kernel (where the buffers live). Of course you can configure your kernel to limit that number (to avoid all of kernel memory going to socket buffers;-).

Answer 3

Since sockets are made available as file descriptors, the limit can be no more than the maximum number of open files, which you can get via

#include <sys/resource.h>

struct rlimit limit;
getrlimit(RLIMIT_NOFILE, &limit);

/* limit.rlim_cur gives the current limit for the process.
   This can be dynamically adjusted via setrlimit, with a maximum
   of limit.rlim_max. They are likely equivalent already though. */
int max_no_files = limit.rlim_cur;

This hard-limit can be adjusted, but how depends on what platform you are on. Linux has /etc/security/limits.conf. I don't know about other platforms.