Socket code failing with large message

Question

Why the following code fails when the returned message from the server is large (over 4000 bytes)?

I get this error:
{"Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."}

It works fine with smaller messages.

What I'm doing basically is call the server until I see the command prompt string, then I stop to process the data...

Code:

Private Function GetData(ByVal command As String, ByRef tcpClient As TcpClient, ByRef stream As NetworkStream) As String

    Dim buffersize As Integer = tcpClient.ReceiveBufferSize
    Dim bytesToRead(buffersize) As Byte
    Dim textReceived As String = ""
    Dim textfromserver As String = ""

    If Not String.IsNullOrEmpty(command) Then
        Try
            SendCommand2Server(command, tcpClient, stream)

            Do
                ' THIS LINE FAILS...'
                Dim numBytesRead As Integer = stream.Read(bytesToRead, 0, bytesToRead.Length)

                If numBytesRead = 0 Then Exit Do

                textfromserver = Encoding.ASCII.GetString(bytesToRead, 0, numBytesRead)
                textReceived += textfromserver

            Loop Until textfromserver.ToLower.Contains(commandprompt)

            Return textReceived

        Catch ex As Exception
            Throw ex
        End Try
    End If

    Return ""

End Function

Any help will be appreciated..

Thanks.