Socket in use error when reusing sockets

Question

I am writing an XMLRPC client in c++ that is intended to talk to a python XMLRPC server.

Unfortunately, at this time, the python XMLRPC server is only capable of fielding one request on a connection, then it shuts down, I discovered this thanks to mhawke's response to my previous query about a related subject

Because of this, I have to create a new socket connection to my python server every time I want to make an XMLRPC request. This means the creation and deletion of a lot of sockets. Everything works fine, until I approach ~4000 requests. At this point I get socket error 10048, Socket in use.

I've tried sleeping the thread to let winsock fix its file descriptors, a trick that worked when a python client of mine had an identical issue, to no avail. I've tried the following

int err = setsockopt(s_,SOL_SOCKET,SO_REUSEADDR,(char*)TRUE,sizeof(BOOL));

with no success.

I'm using winsock 2.0, so WSADATA::iMaxSockets shouldn't come into play, and either way, I checked and its set to 0 (I assume that means infinity)

4000 requests doesn't seem like an outlandish number of requests to make during the run of an application. Is there some way to use SO_KEEPALIVE on the client side while the server continually closes and reopens?

Am I totally missing something?

Answer 1

Do you close the sockets after using it?

Answer 2

Update:

I tossed this into the code and it seems to be working now.

if(::connect(s_, (sockaddr *) &addr, sizeof(sockaddr))) 
  {
    int err = WSAGetLastError();
    if(err == 10048)   //if socket in user error,   force kill and reopen socket
    {
        closesocket(s_);
     WSACleanup();
     WSADATA info;
     WSAStartup(MAKEWORD(2,0), &info);
     s_ = socket(AF_INET,SOCK_STREAM,0);
     setsockopt(s_,SOL_SOCKET,SO_REUSEADDR,(char*)&x,sizeof(BOOL));
    }
  }

Basically, if you encounter the 10048 error (socket in use), you can simply close the socket, call cleanup, and restart WSA, the reset the socket and its sockopt

(the last sockopt may not be necessary)

i must have been missing the WSACleanup/WSAStartup calls before, because closesocket() and socket() were definitely being called

this error only occurs once every 4000ish calls.

I am curious as to why this may be, even though this seems to fix it. If anyone has any input on the subject i would be very curious to hear it

Answer 3

The problem is being caused by sockets hanging around in the TIME_WAIT state which is entered once you close the client's socket. By default the socket will remain in this state for 4 minutes before it is available for reuse. Your client (possibly helped by other processes) is consuming them all within a 4 minute period. See this answer for a good explanation and a possible non-code solution.

Windows dynamically allocates port numbers in the range 1024-5000 (3977 ports) when you do not explicitly bind the socket address. This Python code demonstrates the problem:

import socket
sockets = []
while True:
    s = socket.socket()
    s.connect(('some_host', 80))
    sockets.append(s.getsockname())
    s.close()

print len(sockets)    
sockets.sort()
print "Lowest port: ", sockets[0][1], " Highest port: ", sockets[-1][1]
# on Windows you should see something like this...
3960
Lowest port: 1025  Highest port: 5000

If you try to run this immeditaely again, it should fail very quickly since all dynamic ports are in the TIME_WAIT state.

There are a few ways around this:

1. Manage your own port assignments and use bind() to explicitly bind your client socket to a specific port that you increment each time your create a socket. You'll still have to handle the case where a port is already in use, but you will not be limited to dynamic ports. e.g.

   port = 5000
   while True:
       s = socket.socket()
       s.bind(('your_host', port))
       s.connect(('some_host', 80))
       s.close()
       port += 1
   

2. Fiddle with the SO_LINGER socket option. I have found that this sometimes works in Windows (although not exactly sure why): s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, 1)

3. I don't know if this will help in your particular application, however, it is possible to send multiple XMLRPC requests over the same connection using the multicall method. Basically this allows you to accumulate several requests and then send them all at once. You will not get any responses until you actually send the accumulated requests, so you can essentially think of this as batch processing - does this fit in with your application design?