Connection refused after some time on threaded process in tcp socket requests (c/linux)

Question

I'm trying to make process that takes number of requests each second, on each request new thread is created. Each thread then opens socket connection to address (http port) sends HEAD requests, gets response and closes socket.
Problem I'm having comes when i put more then 3 requests per second, after some time i get error in send() part of function, i keep getting Connection Refused. If I input more requests per second i get errors earlier. If i put only 2 requests per second i don't get errors at all. I suspect that I'm running out of some resource but i can't find which.

Here is basic structure of code

//declarations

socketfd = socket(servinfo->ai_family,servinfo->ai_socktype,servinfo->ai_protocol);

arg = fcntl(socketfd, F_GETFL, NULL)) < 0);
arg |= O_NONBLOCK;
fcntl(socketfd, F_SETFL, arg)

if((conn = connect(socketfd, servinfo->ai_addr, servinfo->ai_addrlen)) < 0)
{
 if(errno == EINPROGRESS)
 {
  do
  {
            tv.tv_sec = CONNECT_TIMEOUT;
   tv.tv_usec = 0;
   FD_ZERO(&myset);
   FD_SET(socketfd, &myset);
   if((res = select(socketfd+1, NULL, &myset, NULL, &tv) > 0)) 
   {
    if( (arg = fcntl(socketfd, F_GETFL, NULL)) < 0) { 
     perror("fcntl get 2");
    } 
    arg &= (~O_NONBLOCK); 
    if( fcntl(socketfd, F_SETFL, arg) < 0) {
     perror("fcntl set 2");
    }
    char szBuf[4096];

    std::string htmlreq = "HEAD / HTTP/1.1\r\nHost:";
    htmlreq += info->hostName;
    htmlreq += "\r\n\r\n";

    if((conn = send(socketfd,htmlreq.c_str(),htmlreq.size(),0)) == -1 && errno != EINTR)
    {
     perror("send");
     close(socketfd);
     return;
    }

    if((conn = recv(socketfd,szBuf,sizeof(szBuf)+1,0)) < 0 && errno != EINTR)
    {
     perror("recv");
     close(socketfd);
     return ;
    }

    close(socketfd);

    // do stuff with data
    break;
   }
   else
   {
                //timeout
    break;
   }
  }while(1);
 }
 else
 {
  perror("connect");
  close(socketfd);
  return; 
 }
}

I removed some error checking from start, what i get as output is "Send: Connection Refused" after some time. I'd appreciate some pointers to what part could be causing problems, platform is ubuntu linux. I'd also be glad to post other parts of code if needed. Tnx in advance.

Answer 1

The resource you're probably running out of is on the server you're connecting to. The connection is being refused by the computer you're connecting to because it's either:

1. Configure to limit the number of connections per second ( based on some criteria )
2. Or the server you're connecting to is under too much load for some reason and can't take any more connections.

Since you always get the error on the third connection it could be that the server you're connecting to limits the number of connections on a per IP basis.

Edit1

You're trying to do a non-blocking connect? Now that I look at it closer it sounds like your problem is with the select, as in select is returning that the socket is readable before it's actually connected and then you're calling send. One of the things to watch out for on non-blocking connects is that the socket becomes both readable and writeable on error. Which means you need to check for both after select returns otherwise you may be missing whatever the actual error is and seeing the send error instead.

This is from Stevens UNP:

FD_ZERO(&rset);
FD_SET(sockfd, &rset);
wset = rset;
tval.tv_sec = nsec;
tval.tv_usec = 0;

if ( (n = Select(sockfd+1, &rset, &wset, NULL,
     nsec ? &tval : NULL)) == 0) {
 close(sockfd);  /* timeout */
 errno = ETIMEDOUT;
 return(-1);
}

if (FD_ISSET(sockfd, &rset) || FD_ISSET(sockfd, &wset)) {
 len = sizeof(error);
 if (getsockopt(sockfd, SOL_SOCKET, SO_ERROR, &error, &len) < 0)
  return(-1);   /* Solaris pending error */
} else
 err_quit("select error: sockfd not set");

done:
Fcntl(sockfd, F_SETFL, flags); /* restore file status flags */

if (error) {
 close(sockfd);  /* just in case */
 errno = error;
 return(-1);
}
return(0);

Answer 2

There are quite a few problems in your code.

First you set the socket to non blocking. I don't understand why you do this. The connect function has an internal timeout and so won't block.

Another problem of your code is that the first if statement will skip the instruction block if the connection immediately succeeds ! Which may happen.

You apparently want to first send the HEAD message. There is no real need to make this one non blocking unless you expect the remote server or the network to be very slow and want a time out on it. In this case the select with non blocking socket would make sens.

Once you send the HEAD message, you expect some data in response that you collect with the recv function. Be aware that this function call may return before the whole data sent is received. You need an independent way to determine that all the data sent has been received. Would the server close the connection ? This would detected by the recv function returning 0.

So the recv should be wrapped into a loop where you append to received data to some buffer or a file and quit when recv returns 0. Use a non blocking socket if you want to add a timeout on this recv operation which may indeed block.

But first try without timeouts to be sure it works at full speed without blocking as your current version.

I suspect the initial connect is slow because of name and IP adresse resolution, and gets faster in subsequent calls because data is cached.