Linux to WinXP over UDP lag

Question

I'm having a problem with a UDPSocket wrapper I've written. I have a high bandwidth low latency local network over which I'm sending UDP packets back and forth. I don't have a concern too much for reliability of packet arrival, but it is incredibly important that the packets that do arrive do so quickly. Here's the relevant code for setting up a socket:

bool UDPSocket::create() {
    int on = 1;
#ifdef WIN32
    if(WSAStartup(MAKEWORD(1,1), &SocketInfo) != 0) {
     MessageBox(NULL, "Cannot initialize WinSock", "WSAStartup", MB_OK);
    }
#endif
    m_sock = socket(PF_INET, SOCK_DGRAM, 0);
#ifdef WIN32
    if(setsockopt(m_sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&on, sizeof(on)) == SOCKET_ERROR)
     return false;
#else
    if(setsockopt(m_sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&on, sizeof(on)) == -1)
        return false;
#endif
    addrLen = sizeof(struct sockaddr);
    return true;
}

bool UDPSocket::bind(const int port) {
    if(!is_valid())
        return false;
    m_addr.sin_family = AF_INET;
    m_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    m_addr.sin_port = htons(port);

    if(::bind(m_sock, (struct sockaddr*)&m_addr, sizeof(m_addr))<0) {
     std::cout << "UDPSocket: error on bind" << std::endl;
        return false;
    }
    return true;
}
bool UDPSocket::send(const std::string s) const {
    const char* buf = s.c_str();

    ::sendto(m_sock, buf, strlen(buf), 0, (const sockaddr*)&clientAddr, addrLen);
    return true;
}

bool UDPSocket::setDestination(const std::string ip, const int port) {
    memset(&clientAddr, 0, sizeof(clientAddr));

    clientAddr.sin_family = AF_INET;
    clientAddr.sin_addr.s_addr = inet_addr(ip.c_str());
    clientAddr.sin_port = htons(port);

    return true;
}

int UDPSocket::recv(std::string& s) const {
    char buffer[MAXRECV + 1];
    struct timeval tv;
    fd_set fdset;
    int rc, nread;

    memset(&buffer, 0, sizeof(buffer));

    FD_ZERO(&fdset);
    FD_SET(m_sock, &fdset);

    tv.tv_sec = 0;
    tv.tv_usec = m_timeout;

    rc = select(m_sock + 1, &fdset, (fd_set *) 0, (fd_set *) 0, &tv);

    if(FD_ISSET(m_sock, &fdset)) {
#ifdef WIN32
        nread = ::recvfrom(m_sock, buffer, MAXRECV, 0, (sockaddr*)&clientAddr, const_cast< int * __w64 >(&addrLen));
#else
     nread = ::recvfrom(m_sock, buffer, MAXRECV, 0, (sockaddr*)&clientAddr, (socklen_t*)&addrLen);
#endif
     if(nread < 0) {
      return -1;
     } else if(nread == 0) {
      return 0;
     }
     s = std::string(buffer);
     return nread;
    } else {
     return 0;
    }
}
void UDPSocket::set_non_blocking(const bool b) {
    mNonBlocking = b;
#ifdef WIN32
    u_long argp = b ? 1 : 0;
    ioctlsocket(m_sock, FIONBIO, &argp);
#else
    int opts = fcntl(m_sock, F_GETFL);
    if(opts < 0) return;
    if(b)
        opts |= O_NONBLOCK;
    else
        opts &= ~O_NONBLOCK;

    fcntl(m_sock, F_SETFL, opts);
#endif
}

My user code, on both ends, create a "sending" and "receiving" UDPSocket and bind them to their respective ports, then use send() to send data and recv() to receive. On one hand, the linux side seems to receive practically immediately, but the Windows side has a delay up to 1 second before receiving data. However, ::recv() never returns 0 in this time. Am I missing something obvious?

Answer 1

have you tried all four combinations (linux->linux, win->linux, linux->win, win->win)? which have delays and which not?

also, use a packet sniffer (like tcpdump or wireshark) to see if the delay is before or after hitting the wire.

Answer 2

It's not a direct answer. May be you can try tools like ttcp (http://www.pcausa.com/Utilities/pcattcp.htm), which can both test tcp and udp performance. And, you may also check their source code, which is in public domain.

Answer 3

ah..let me guess. I see in your code that you call your wrapper "recv" function and then run a select.

A coupe of questions.

a) Do you have your wrapper "recv" in a loop.

b) Is your Select timeout value 1000 ms.

c) How do you decide when to send?

d) Do you start sending from Windows?

I have a fair idea whats causing your problem. However, to be sure, I need to understand at what point do you decide to call your wrapper "recv" and at what point you call "send".

Answer 4

If you've got the time to experiment, try an IOCP version of the code.

I never trusted the win select implementation, with its 64 socket limit.