Socket.SendAsync taking several seconds to complete.

Question

I'm trying to optimize a tcp socket wrapper which is struggling with lots of inbound connections. I'm testing it within a basic chat server and a small client app to send clients to it. Both apps are on a separate W2k3 server connected by a gigabit switch.

By trial and error I've refined my testing to 10 clients connecting at 100ms intervals, then once all 10 are connected they each send one "enter room" message to the server, again at 100ms intervals. When the server recieves a message it replies to the sender with a list of everyone in the room, and also sends a message to everyone else in the room to say theres a new arrival.

Each send is taking over 1 second to complete (this goes up to 3-4 seconds with 100+ clients), and with logging I've established that the delay is between the Socket.SendAync and the corresponding event being raised. Cpu usage remains low throughout.

I've tried everything I can think of and spent days looking for clues online and I'm at a complete loss. This cant be normal can it?

Edit: Code as requested. I've tidied it up a bit, removed non-relevant counters and logging etc, it currently has hack on top of kludge on top of hack as I try to narrow down the issue.

    private void DoSend(AsyncUserToken token, String msg)
    {
        SocketAsyncEventArgs writeEventArgs = new SocketAsyncEventArgs();
        writeEventArgs.Completed += ProcessSend;
        writeEventArgs.UserToken = token;
        Byte[] sendBuffer = Encoding.UTF8.GetBytes(msg + LineTerminator);
        writeEventArgs.SetBuffer(sendBuffer, 0, sendBuffer.Length);

        Interlocked.Add(ref m_totalBytesAttemptedSend, sendBuffer.Length);
        Logger2.log(Logger2.Debug5, token.ConnectionId, "Total Bytes attempted: " + m_totalBytesAttemptedSend);

        bool willRaiseEvent = true;
        try
        {
            willRaiseEvent = token.Socket.SendAsync(writeEventArgs);
        }
        catch (Exception e)
        {
            Logger2.log(Logger2.Debug2, token.ConnectionId, e.Message);
            writeEventArgs.Dispose();
        }
        if (!willRaiseEvent)
        {
            ProcessSend(null, writeEventArgs);
        }
    }


    private void ProcessSend(Object sender, SocketAsyncEventArgs e)
    {
        AsyncUserToken token = (AsyncUserToken)e.UserToken;
        Logger2.log(Logger2.Debug5, token.ConnectionId, "Send Complete");
        if (e.SocketError == SocketError.Success)
        {
            Interlocked.Add(ref m_totalBytesSent, e.BytesTransferred);
            Logger2.log(Logger2.Debug5, ((AsyncUserToken)e.UserToken).ConnectionId, "Total Bytes sent: " + m_totalBytesSent);

        }
        else
        {
            if (token.Connected)
            {
                CloseClientSocket(token);
            }
        }
        e.Dispose();
    }

Answer 1

How much data are you sending on each connection and how fast are you sending it?

Usually the reason that async sends take a long time to complete is that you have filled the TCP window (see here for details) and the local TCP stack is unable to send any more data until it gets some ACKs from the peer. If you continue to send data then you are simply queueing it locally in the networking subsystem as the stack isn't allowed to send it. This can be made worse with congestion and packet loss as the data in transit takes longer to get to the peer and the ACKs take longer to get back...

If this is the case (and a tool such as WireShark should enable you to see the window size updates, and zero window situations, etc) then increasing the connection's window size might help (see here).

If your protocol doesn't have explicit flow control then the above situation is even more likely to occur. It's better to include some form of explicit flow control, IMHO, in your protocol design to avoid this.

It's also sensible to include some send side protection from this as an unbounded sender can chew through memory very quickly. One approach that I've used that works well is to have a queue that outbound data can be placed in and data from this queue is actually sent based on the send completions of earlier sends (see here).