read and write to same socket (TCP) using select

Question

We're writing a client and a server to do (what I thought was) pretty simple network communications. Mulitple clients connect to the server which then is supposed to send the data back to all other clients.

The server just sits in a blocking select loop waiting for traffic, and when it comes, sends the data to the other clients. This seems to work just fine.

The problem is the client. In response to a read, it will sometimes want to do a write.

However, I've found that if I use:

 rv = select(fdmax + 1, &master_list, NULL, NULL, NULL);

My code will block until there is new data to read. But sometimes (asynchronously, from another thread) I'll have new data to write on the network communication thread. So, I want my select to wake up periodically and let me check if there are data to write, like:

if (select(....) != -1)
{
  if (FD_SET(sockfd, &master_list))
     // handle data or disconnect
  else
     // look for data to write and write() / send() those.
}

I tried setting the select to poll mode (or ridiculously short timeouts) with:

// master list contains the sockfd from the getaddrinfo/socket/connect seq
struct timeval t;
memset(&t, 0, sizeof t);
rv = select(fdmax + 1, &master_list, NULL, NULL, &t);

but have found that then then client never gets any incoming data.

I also tried setting the socket fd to be non-blocking, like:

fcntl(sockfd, F_SETFL, O_NONBLOCK);

but that doesn't solve the problem:

1. if my client select() has no struct timeval, reading data works, but it never unblocks to let me look for writable data.
2. if my client select() has a timeval to get it to poll, then it never signals that there are incoming data to read, and my app freezes thinking there is no network connection made (despite the fact that all other function calls have succeeded)

Any pointers at all as to what I could be doing wrong? Is it not possible to do read-write on the same socket (I can't believe that to be true).

(EDIT: The correct answer, and thing that I remembered on the server but not on the client, is to have a second fd_set, and copy master_list before each call to select():

// declare and FD_ZERO read_fds:
// put sockfd in master_list

while (1)
{
   read_fds = master_list;
   select(...);

   if (FD_ISSET(read_fds))
     ....
   else
     // sleep or otherwise don't hog cpu resources
}

)

Answer 1

Everything looks fine, except the line where you do if (FD_SET(sockfd, &master_list)). I have a very similar code structure and I used FD_ISSET. You're supposed to test if the list is set, not to set it again. Other than that, I see nothing else.

Edit. Also, I have the following for the timeout:

timeval listening_timeout;
listening_timeout.tv_sec = timeout_in_seconds;
listening_timeout.tv_usec = 0;

perhaps there's an issue if you set it to 0 (as you seem to be doing?)

Edit2. I remembered I ran into a strange problem when I wasn't clearing the read set after the select exited and before I entered it again. I had to do something like:

FD_ZERO(&sockfd);
FD_SET(sockfd, &rd);

before I was entering select. I can't remember why though.

Answer 2

I seem to recall a trick about creating and sharing a read/write filedescriptor between the network thread and the main thread that is added to the descriptors in the select call. This fd has one byte written to it by the main thread when it has something to send. The write wakes up network thread from the select call and the network thread is then grabs the data from a shared buffer and writes it to the network then go back to sleep in the select.

Sorry if that's a bit vague and lacking of code ... and my memory may be incorrect.. so others may have to guide you further.

Answer 3

I don't see anything wrong with your code, so it should work. If you can't get it working, one way to work around it would be to create a pipe to be used by your reading thread and the thread that prepares things for writing, and add the pipe's reading end to your select set. Then, when the other thread has prepared data to write, it just sends something on the pipe, your reading thread gets woken up from the select, and it can then do the writing. Depending on how often there is data to read or write, this could also be more efficient.

Answer 4

2 threads should be able to work with the same socket at a time, so your main thread should be able to write to a client while the other one sleeps in select waiting for incoming data. This of course assumes that both threads have access to the client list.