What does a working example of dealing with packet fragmentation in C# look like when using the Begin* End* method?

Question

After enough playing with asynchronous socket programming I noticed that the server was receiving chunked payloads (ie: more than one complete payload sitting in the same buffer). So I came up with the following:

if (bytes_to_read > 0)
{
    while (bytes_to_read > 0)

        // Get payload size as int.

        // Get payload in byte format.

        // Do something with payload.

        // Decrease the amount of bytes to read.
    }

    // Wait for more data.
}

And then I noticed packet fragmentation (ie: what I thought were complete payloads chunked together wasn't always so) which changed the previous code to something like:

if (bytes_to_read > 0)
{
    while (bytes_to_read > 0)
    {
        // Get payload size as int.

        // Check if the payload size is less than or equal to the amount of bytes left to read.
        if (payload_size <= bytes_to_read)
        {
            // Get payload in byte format.

            // Do something with payload.

            // Decrease the amount of bytes to read.
        }
        else
        {
            // We received a fragmented payload.
            break;
        }
    }

    if (bytes_to_read == 0)
    {
        // Wait for more data.
    }
    else if (bytes_to_read > 0)
    {
        // Wait for more data where we left off. ***
    }
    else
    {
        // Something awful happened.
    }
}

*** I don't even know how to go about this and would like to see code for it. I had an idea that it involved copying the in-completed payload to the beginning of the buffer and then picking up from there.

The pseudo code I included is based on the Begin* End* method I am using (I'm aware that I should be using the *Async set of methods found here -> http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasynceventargs.aspx <- but I think my overall question still applies).

I am seeking the answers to 2 questions--namely:

1. Is this approach correct or am I missing something?
2. What does a working example of dealing with packet fragmentation in C# look like?

EDIT: I'm using raw sockets.

Thanks in advance for all your help.

EDIT: John Saunders and Greg Hewgill have brought up the point of treating data as a stream but that does not provide me with a concrete example of how to deal with the last chunked payload sometimes being fragmented.

EDIT: I have read Jon Skeet's answer here which is basically along the same lines as the other answers I have seen but it doesn't help me much as I already get what I have to do but not how to do it.

EDIT: To elaborate on what I mean by fragmentation, consider the following the receive buffers:

• 224TEST3foo3bar
• 224TEST3foo3bar224TEST3foo3bar
• 224TEST3foo3bar224TEST3foo
• 3bar224TEST3foo3bar

EDIT: I found this and this which lead me here. Vadym Stetsiak has cleared nearly everything up (his was one of the answers I was looking for).

Answer 1

This may or may not have anything to do with fragmentation.

In general, the socket will pass you as many bytes at a time as it feels like. Your job is to know how many bytes are in your overall message, and to read them all. Just keep looping until you have all the bytes you need, or until there's an exception.

---

The following code is untested right now. I thought I'd post it before writing the server side of it and testing both.

private static string ReceiveMessage(Socket socket)
{
    const int BUFFER_SIZE = 1024;
    var inputBuffer = new byte[BUFFER_SIZE];
    var offset = 0;
    var bytesReceived = socket.Receive(
        inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
    if (bytesReceived < 2)
    {
        throw new InvalidOperationException("Receive error");
    }

    var inputMessageLength = inputBuffer[0]*256 + inputBuffer[1];
    offset += bytesReceived;
    var totalBytesReceived = bytesReceived;
    while (bytesReceived > 0 &&
           totalBytesReceived < inputMessageLength + 2)
    {
        bytesReceived = socket.Receive(
            inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
        offset += bytesReceived;
        totalBytesReceived += bytesReceived;
    }

    return Encoding.UTF8.GetString(
        inputBuffer, 2, totalBytesReceived - 2);
}

Note that the receipt of the message length is wrong. The socket layer could give it to me a byte at a time. I'm going to revisit that as part of a refactoring that will receive the count into a separate two-byte buffer, and change the loop into a single do/while.

Answer 2

When you have to do it yourself, it can be done like so (reference here):

/// 
/// Server state holds current state of the client socket
///
class AsyncServerState
{
   public byte[] Buffer = new byte[512]; //buffer for network i/o
   public int DataSize = 0; //data size to be received by the server

   //flag that indicates whether prefix was received
   public bool DataSizeReceived = false;

   public MemoryStream Data = new MemoryStream(); //place where data is stored
   public SocketAsyncEventArgs ReadEventArgs = new SocketAsyncEventArgs();
   public Socket Client;
}

/// 
/// Implements server receive logic
/// 
private void ProcessReceive(SocketAsyncEventArgs e)
{
    //single message can be received using several receive operation
    AsyncServerState state = e.UserToken as AsyncServerState;

    if (e.BytesTransferred <= 0 || e.SocketError != SocketError.Success)
    {
        CloseConnection(e);
    }

    int dataRead = e.BytesTransferred;
    int dataOffset = 0;
    int restOfData = 0;

    while (dataRead > 0)
    {
        if (!state.DataSizeReceived)
        {
            //there is already some data in the buffer
            if (state.Data.Length > 0)
            {
                restOfData = PrefixSize - (int)state.Data.Length;
                state.Data.Write(state.Buffer, dataOffset, restOfData);
                dataRead -= restOfData;
                dataOffset += restOfData;
            }
            else if (dataRead >= PrefixSize)
            {   //store whole data size prefix
                state.Data.Write(state.Buffer, dataOffset, PrefixSize);
                dataRead -= PrefixSize;
                dataOffset += PrefixSize;
            }
            else
            {   // store only part of the size prefix
                state.Data.Write(state.Buffer, dataOffset, dataRead);
                dataOffset += dataRead;
                dataRead = 0;
            }

            if (state.Data.Length == PrefixSize)
            {   //we received data size prefix
                state.DataSize = BitConverter.ToInt32(state.Data.GetBuffer(), 0);
                state.DataSizeReceived = true;

                state.Data.Position = 0;
                state.Data.SetLength(0);
            }
            else
            {   //we received just part of the headers information
                //issue another read
                if (!state.Client.ReceiveAsync(state.ReadEventArgs))
                    ProcessReceive(state.ReadEventArgs);
                    return;
            }
        }

        //at this point we know the size of the pending data
        if ((state.Data.Length + dataRead) >= state.DataSize)
        {   //we have all the data for this message

            restOfData = state.DataSize - (int)state.Data.Length;

            state.Data.Write(state.Buffer, dataOffset, restOfData);
            Console.WriteLine("Data message received. Size: {0}",
                                  state.DataSize);

            dataOffset += restOfData;
            dataRead -= restOfData;

            state.Data.SetLength(0);
            state.Data.Position = 0;
            state.DataSizeReceived = false;
            state.DataSize = 0;

            if (dataRead == 0)
            {
                if (!state.Client.ReceiveAsync(state.ReadEventArgs))
                    ProcessReceive(state.ReadEventArgs);
                    return;
            }
            else
                continue;
        }
        else
        {   //there is still data pending, store what we've
            //received and issue another BeginReceive
            state.Data.Write(state.Buffer, dataOffset, dataRead);

            if (!state.Client.ReceiveAsync(state.ReadEventArgs))
                ProcessReceive(state.ReadEventArgs);

            dataRead = 0;
        }
    }
}

I did not do it exactly this way myself but it helped.