Sending large serialized objects over sockets is failing only when trying to grow the byte Array, but ok when using a massive byte array

Question

I have code where I am trying to grow the byte array while receiving the data over my socket. This is erroring out.

    public bool ReceiveObject2(ref Object objRec, ref string sErrMsg)
    {
        try
        {
            byte[] buffer = new byte[1024];
            byte[] byArrAll = new byte[0];
            bool bAllBytesRead = false;

            int iRecLoop = 0;

            // grow the byte array to match the size of the object, so we can put whatever we 
            // like through the socket as long as the object serialises and is binary formatted 
            while (!bAllBytesRead)
            {
                if (m_socClient.Receive(buffer) > 0)
                {
                    byArrAll = Combine(byArrAll, buffer);
                    iRecLoop++;
                }
                else
                {
                    m_socClient.Close();
                    bAllBytesRead = true;
                }
            }

            MemoryStream ms = new MemoryStream(buffer);
            BinaryFormatter bf1 = new BinaryFormatter();
            ms.Position = 0;
            Object obj = bf1.Deserialize(ms);
            objRec = obj;

            return true;
        }
        catch (System.Runtime.Serialization.SerializationException se)
        {
            objRec = null;
            sErrMsg += "SocketClient.ReceiveObject " + "Source " + se.Source + "Error : " + se.Message;
            return false;
        }
        catch (Exception e)
        {
            objRec = null;
            sErrMsg += "SocketClient.ReceiveObject " + "Source " + e.Source + "Error : " + e.Message;
            return false;
        }
    }

    private byte[] Combine(byte[] first, byte[] second)
    {
        byte[] ret = new byte[first.Length + second.Length];
        Buffer.BlockCopy(first, 0, ret, 0, first.Length);
        Buffer.BlockCopy(second, 0, ret, first.Length, second.Length);
        return ret;
    }    

Error: mscorlibError : The input stream is not a valid binary format. The starting contents (in bytes) are: 68-61-73-43-68-61-6E-67-65-73-3D-22-69-6E-73-65-72 ...

Yet when I just cheat and use a MASSIVE buffer size its fine.

        public bool ReceiveObject(ref Object objRec, ref string sErrMsg)
    {
        try
        {
            byte[] buffer = new byte[5000000];

            m_socClient.Receive(buffer);
            MemoryStream ms = new MemoryStream(buffer);
            BinaryFormatter bf1 = new BinaryFormatter();

            ms.Position = 0;
            Object obj = bf1.Deserialize(ms);
            objRec = obj;

            return true;
        }
        catch (Exception e)
        {
            objRec = null;
            sErrMsg += "SocketClient.ReceiveObject " + "Source " + e.Source + "Error : " + e.Message;
            return false;
        }
    }

This is really killing me. I don't know why its not working. I have lifted the Combine from a suggestion on here too, so I am pretty sure this is not doing the wrong thing?

I hope someone can point out where I am going wrong

Answer 1

I'm not really familiar with C# networking, but aren't you appending full 1024 bytes to the buffer every time you call Combine() and ignoring the number of bytes read off the socket? You probably need at least one extra parameter to that function telling how many bytes to copy from second.

Answer 2

That Combine method is a really expensive way to grow an array, especially when MemoryStream is designed to address this; and the other replies are right: you must check the number of bytes read:

using(MemoryStream ms = new MemoryStream()) {
    int bytesRead;
    while((bytesRead = m_socClient.Receive(buffer)) > 0) {
        ms.Write(buffer, 0, bytesRead);
    }
    // access ms.ToArray() or ms.GetBuffer() as desired, or
    // set Position to 0 and read
}

Of course, you could just read directly from the stream (pass it to your reader)

Also - if your serialization is too big, you might consider alternative encoders, such as protobuf-net (although this will change the code a little). This may fix the problem with a huge object.