Weird send() problem (with Wireshark log)

Question

I had another question about this issue, but I didn't ask properly, so here I go again!

I'm sending a file by sending it in chunks. Right now, I'm playing around with different numbers for the size of that chunk, to see what size is the most efficient.

When testing on the localhost, any chunk size seems to work fine. But when I tested it over the network, it seems like the maximum chunk size is 8191 bytes. If I try anything higher, the transfer becomes extremely, painfully, slow.

To show what happens, here are the first 100 lines of Wireshark logs when I use a chunk size of 8191 bytes, and when I use a chunk size of 8192 bytes: (the sender is 192.168.0.102, and the receiver is 192.168.0.100)

8191: http://pastebin.com/E7jFFY4p

8192: http://pastebin.com/9P2rYa1p

Notice how in the 8192 log, on line 33, the receiver takes a long time to ACK the data. This happens again on line 103 and line 132. I believe this delay is the root of the problem.

Note that I have not modified the SO_SNDBUF option nor the TCP_NODELAY option.

So my question is, why am I getting delayed ACKs when sending files in chunks of 8192 bytes, when everything works fine when using chunks of 8191 bytes?

Answer 1

Check your "Flow Control" setting on your NICs and switches. If it is on, it may be the cause of your problem.

And for proper dissection you will need to run wireshark on both ends of the transfer.

Answer 2

I figured it out! First by myself, and then after some more digging, I found this: http://support.microsoft.com/kb/823764

What was actually happening is that because the send buffer allocated by Winsock is by default (on my machine) exactly 8192 bytes, when I put that amount of bytes in the buffer (effectively completely filling it up), the next send() will give WSAEWOULDBLOCK. Then, I would only receive the next FD_WRITE once the bytes have been ACKed.

But at the same time, the receiving machine was not sending an ACK because of the delayed ACK algorithm. This put the transmission into a deadlock for 200 ms, after which the receiving machine finally ACKs the data, which then allows the send function to receive an FD_WRITE.

Of course, all this does not happen when I used 8191 bytes because I did not fill up the whole buffer, and thus the next send() did not block. This meant that Winsock would always keep sending data so that the delayed ACK algorithm never kicked in on the receiving end (except on the last packet if it is an odd-numbered packet).

Hope this helps anyone else with the same issue I had.