Python sock.listen(...)

Question

All the examples I've seen of sock.listen(5) in the python documentation suggest I should set the max backlog number to be 5. This is causing a problem for my app since I'm expecting some very high volume (many concurrent connections). I set it to 200 and haven't seen any problems on my system, but was wondering how high I can set it before it causes problems..

Anyone know?

Edit: Here's my accept() loop.

while True:    
    try:
        self.q.put(sock.accept())
    except KeyboardInterrupt:
        break
    except Exception, e:
        self.log("ERR %s" % e)

Answer 1

The doc say this

socket.listen(backlog) Listen for connections made to the socket. The backlog argument specifies the maximum number of queued connections and should be at least 1; the maximum value is system-dependent (usually 5).

Obviously the system value is more than 5 on your system. I don't see why setting it to a larger number would be a problem. Perhaps some memory is reserved for each queued connection.

My linux man page has this to say

If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; the default value in this file is 128. In kernels before 2.4.25, this limit was a hard coded value, SOMAXCONN, with the value 128.

Answer 2

You don't need to adjust the parameter to listen() to a larger number than 5.

The parameter controls how many non-accept()-ed connections are allowed to be outstanding. The listen() parameter has no bearing on the number of concurrently connected sockets, only on the number of concurrent connections which have not been accept()-ed by the process.

If adjusting the parameter to listen() has an impact on your code, that is a symptom that too much delay occurs between each call to accept(). You would then want to change your accept() loop such that it has less overhead.

In your case, I am guessing that self.q is a python queue, in which case you may want to call self.q.put_nowait() to avoid any possibility of blocking the accept() loop at this call.