Avoid reusing of the same fd number in a multithread socket application

Question

I have an asynchronous application executing several threads doing operations over sockets where operations are scheduled and then executed asynchronously.

I'm trying to avoid a situation when once scheduled a read operation over a socket, the socket gets closed and reopened(by possibly another peer in another operation), before the first operation started execution, which will end up reading the proper file descriptor but the wrong peer.

The problem comes because (accept();close();accept()) returns the same fd in both accepts() which can lead to the above situation.

I can't see a way of avoiding it.

any hint?

Answer 1

How do you manage the sockets? It sounds like you have multiple threads any of which can:

1. accept an incoming connection
2. close an existing connection
3. make a new outgoing connection

It sounds like you need a way to mediate access to the various sockets floating around. Have you considered associating each socket with mutex which prevents closing the socket while it's still in use, or maybe putting each socket descriptor in a struct with an atomic reference count which will prevent other threads from closing it until all threads are done using it?

Answer 2

a socket is a 5-tuple {local-addr,local-port,remote-addr,remote-port,proto}, so if you are able to use these properties instead of fd for event/handler routing you can avoid the fd clash.

another option would be to serialize all close()/accept() operations (priorities ?) so that they cannot intermix

Answer 3

Keep a count of the pending operations (read/write, etc.) for each socket and also whether there is a pending close request on the socket. Where before you would have called close, check first whether there are any pending operations. If there are, call shutdown instead, and then only call close when the pending operations count reaches 0.

Answer 4

Great question! I didn't even realize that such a problem could occur.

The only answer that I can think of is that you musn't use close() to signal that a socket is terminated. One solution is to use shutdown() to terminate the connection. You could then close() the socket safely by employing reference counting.

Answer 5

Ok, found the answer.

The best way here is to call accept() and get the lowest fd available, duplicate it with a number known by you like dup2(6,1000) and close(6), you have now control of the fd range you use.

Next accept will come again with 6 or similar, and we'll dup2(6,999); and keep decreasing like that and reseting it if it gets too low.

Since the accepting is done always in the same thread and dup2 and close aren't expensive compared to accept which is always done there it's perfect for my needs.

Answer 6

I would still be careful of using dup2() to a well-known fd value. Remember that dup2() will perform a close on the target before duping; that could conflict with some unrelated thread doing unrelated I/O if you start to have 1000 files open.

If I were you, given the constraints you're insisting upon, I would use dup() (not dup2()) inside of a mutex. (Maybe per-fd mutexes if you're that concerned about it.)