Buffer management for socket application best practice

Question

Having a Windows IOCP app............

I understand that for async i/o operation (on network) the buffer must remain valid for the duration of the send/read operation.

So for each connection I have one buffer for the reading.
For sending I use buffers to which I copy the data to be sent. When the sending operation completes I release the buffer so it can be reused.

So far it's nice and not of a big issue.

What remains unclear is how do you guys do this?

Another thing is that even when having things this way, I mean multi-buffers, the receiver side might be flooded (talking from experience) with data.
Even setting SO_RCVBUF to 25MB didn't help in my testings.
So what should I do? Have a to-be-sent queue?

Answer 1

I reference count the per connection (socket) and per operation (buffer) structures. This works very well and deals with the lifetime issues perfectly. Each time an overlapped operation is posted the reference count of the per connection is incremented and a new buffer is allocated from the pool. When the operation completes I process the results and release the reference on the socket and the buffer. If this is the last reference then the structure is cleaned up (buffers go back to the pool, etc).

You can see all of this in action in my free IOCP client/server framework which is available for download from here.