Non-blocking TCP buffer issues.

Question

Hi!

I think I'm in a problem. I have two TCP apps connected to each other which use winsock I/O completion ports to send/receive data (non-blocking sockets).

Everything works just fine until there's a data transfer burst. The sender starts sending incorrect/malformed data.

I allocate the buffers I'm sending on the stack, and if I understand correctly, that's a wrong to do, because these buffers should remain as I sent them until I get the "write complete" notification from IOCP.

Take this for example:

void some_function()
{
    char cBuff[1024];

    // filling cBuff with some data

    WSASend(...); // sending cBuff, non-blocking mode

    // filling cBuff with other data

    WSASend(...); // again, sending cBuff

    // ..... and so forth!
}

If I understand correctly, each of these WSASend() calls should have its own unique buffer, and that buffer can be reused only when the send completes.
Correct?

Now, what strategies can I implement in order to maintain a big sack of such buffers, how should I handle them, how can I avoid performance penalty, etc'?
And, if I am to use buffers that means I should copy the data to be sent from the source buffer to the temporary one, thus, I'd set SO_SNDBUF on each socket to zero, so the system will not re-copy what I already copied. Are you with me? Please let me know if I wasn't clear.

Answer 1

I don't think it is a good idea to do a second send before the first send is finished.

Similarly, I don't think it is a good idea to change the buffer before the send is finished.

I would be inclined to store the data in some sort of queue. One thread can keep adding data to the queue. The second thread can work in a loop. Do a send and wait for it to finish. If there is more data do another send, else wait for more data.

You would need a critical section (or some such) to nicely share the queue between the threads and possibly an event or a semaphore for the sending thread to wait on if there is no data ready.

Answer 2

Now, what strategies can I implement in order to maintain a big sack of such buffers, how should I handle them, how can I avoid performance penalty, etc'?

It's difficult to know the answer without knowing more about your specific design. In general I'd avoid maintaining your own "sack of buffers" and instead use the OS's built in sack of buffers - the heap.

But in any case, what I would do in the general case is expose an interface to the callers of your code that mirror what WSASend is doing for overlapped i/o. For example, suppose you are providing an interface to send a specific struct:

struct Foo
{
   int x;
   int y;
};

// foo will be consumed by SendFoo, and deallocated, don't use it after this call
void SendFoo(Foo* foo);

I would require users of SendFoo allocate a Foo instance with new, and tell them that after calling SendFoo the memory is no longer "owned" by their code and they therefore shouldn't use it.

You can enforce this even further with a little trickery:

// After this operation the resultant foo ptr will no longer point to
// memory passed to SendFoo
void SendFoo(Foo*& foo);

This allows the body of SendFoo to send the address of the memory down to WSASend, but modify the passed in pointer to NULL, severing the link between the caller's code and their memory. Of course, you can't really know what the caller is doing with that address, they may have a copy elsewhere.

This interface also enforces that a single block of memory will be used with each WSASend. You are really treading into more than dangerous territory trying to share one buffer between two WSASend calls.

Answer 3

Take a serious look at boost::asio. Asynchronous IO is its specialty (just as the name suggests.) It's pretty mature library by now being in Boost since 1.35. Many people use it in production for very intensive networking. There's a wealth of examples in the documentation.

One thing for sure - it take working with buffers very seriously.

Edit:

Basic idea to handling bursts of input is queuing.

• Create, say, three linked lists of pre-allocated buffers - one is for free buffers, one for to-be-processed (received) data, one for to-be-sent data.
• Every time you need to send something - take a buffer off the free list (allocate a new one if free list is empty), fill with data, put it onto to-be-sent list.
• Every time you need to receive something - take a buffer off the free list as above, give it to IO receive routine.
• Periodically take buffers off to-be-sent queue, hand them off to send routine.
• On send completion (inline or asynchronous) - put them back onto free list.
• On receive completion - put buffer onto to-be-processed list.
• Have your "business" routine take buffers off to-be-processed list.

The bursts will then fill that input queue until you are able to process them. You might want to limit the queue size to avoid blowing through all the memory.