Sending large objects over TCP: "End of Stream encountered before parsing was completed"

Question

I keep getting an SerializationException whenever I try to deserialize a list of doubles from a NetworkStream:

End of Stream encountered before parsing was completed

I have a simple client server architecture: my TcpTransportClient wraps the functionality of a TcpClient and I utilize two basic methods: Send (sends a message) and Receive (blocks until a message is received).

The Send function takes a Message, serializes it using a BinaryFormatter and sends the bytes via the NetworkStream.

public void Send(Message message)
{
    if (message == null)
    {
        throw new TransportException("Invalidate Parameter In Send Call");
    }

    if (_tcpClient == null || !_tcpClient.Connected)
    {
        throw new TransportException("Client Not Connected");
    }

    lock (_sync)
    {
        // Serialzie the message
        MemoryStream memStream = new MemoryStream();
        _serializer.Serialize(memStream, message);

        // Get the bytes of of the memory stream
        byte[] buffer = memStream.GetBuffer();

        // Write the message to the network stream
        NetworkStream networkStream = _tcpClient.GetStream();
        networkStream.Write(buffer, 0, buffer.Length);
        networkStream.Flush();
    }
}

The receive function reads bytes into a buffer from the NetworkStream, then deserializes the message by using a BinaryFormatter:

public Message Receive()
{
    if (_tcpClient == null || !_tcpClient.Connected)
    {
        throw new TransportException("Client Not Connected");
    }

    byte[] buffer;
    MemoryStream memStream = new MemoryStream();
    NetworkStream netStream = _tcpClient.GetStream();
    try
    {
        do
        {
            // Allocate a new buffer
            buffer = new byte[_tcpClient.ReceiveBufferSize];

            // Read the message buffer
            int sizeRead = netStream.Read(buffer, 0, buffer.Length);

            // Write the buffer to the memory stream
            memStream.Write(buffer, 0, sizeRead);

        } while (netStream.DataAvailable);

        // Reset the memory stream position
        memStream.Position = 0;

        // Deserialize the message
        object tmp = _deserializer.Deserialize(memStream); // <-- The exception is here!

        // Cast the object to a message
        return (Message)tmp;
    }
    catch (System.Exception e)
    {
        if (_tcpClient == null || !_tcpClient.Connected)
        {
            throw new TransportException("Client Not Connected");
        }
        else
        {
            throw e;
        }
    }
}

I have a basic sending thread:

TcpTransportClient client = new TcpTransportClient(GetLocalAddress(), servicePort);
client.Connect();

Thread sendThread = new Thread(() =>
{
    List<double> dataSet = new List<double>();
    for (double d = 0.0; d < 500.0; d++)
    {
        dataSet.Add(d);
    }

    while (true)
    {
        try
        {
            // Serialize the data set
            MemoryStream memStream = new MemoryStream();
            BinaryFormatter binFormat = new BinaryFormatter();
            binFormat.Serialize(memStream, (object)dataSet);

            // Create a message
            Message msg = new Message();

            // Hold the object bytes in an opaque field
            msg.SetFieldValue(1000, memStream.GetBuffer());

            // Set the message topic
            msg.SetTopic("client.dataSet");

            // Send the message
            client.Send(msg);

            // Sleep
            Thread.Sleep(3000);
        }
        catch (TransportException)
        {
            break;
        }
        catch(Exception)
        {
            //ignore it
        }
    }
});
sendThread.IsBackground = true;
sendThread.Start();

And a receive thread which gets started whenever a TcpClient is accepted:

public void HandleAcceptedClient(TcpTransportClient client)
{
    Thread receiveThread = new Thread(() =>
    {
        while (true)
        {
            try
            {
                Message msg = client.Receive();
                Trace.WriteLine("Server Received: " + msg.GetTopic());
                byte[] buffer = msg.GetFieldOpaqueValue(1000);
                MemoryStream memStream = new MemoryStream(buffer);
                BinaryFormatter binFormat = new BinaryFormatter();

                List<double> dataSet = (List<double>)binFormat.Deserialize(memStream);
                if (dataSet.Count == 500)
                {
                    Trace.WriteLine(DateTime.Now + ": OK");
                }
                else
                {
                    Trace.WriteLine(DateTime.Now + ": FAIL");
                }
            }
            catch (TransportException)
            {
                break;
            }
            catch(Exception)
            {
                // ignore it
            }

        }

    });
    receiveThread.IsBackground = true;
    receiveThread.Start();
}

The exception always occurs when I try to deserialize the message in Receive method of my TcpTransportClient, but the problem only occurs if I put some data in the data set. What's the proper way to send a list of values over a network and successfully deserialize them on the receiving end?

P.S. I tried the solution in a nearly identical question, but it didn't work: I'm still getting the same exception.

Answer 1

while (netStream.DataAvailable);

That's not correct. You should stop reading when the Read() call returns 0. The DataAvailable property just tells you whether or not the Read() call will block, waiting to allow the server to catch up.

Answer 2

You need message framing.