How to reliably tell when finished reading from a socket in Java?

Question

We are integrating with an external product that requires us to communicate with it using Java sockets. We've been able to read small responses from the server with no issues, but larger responses are causing some headaches.
I made some changes to the socket handling logic and now we seem to be able to read large responses 90% of the time. It does still fail occasionally. Failure, in this case, means that the Java client stops reading from the socket before the entire response has been read. The client thinks that the read operation is finished, and stops normally - there are no exceptions or timeouts involved.
Here's what the current logic looks like:

StringWriter response = new StringWriter();
PrintWriter writer = new PrintWriter(response);

char[] buf = new char[4096];
int readChars;
do {
   readChars = responseBufferedReader.read(buf);
   writer.write(buf, 0, readChars);
} while(readChars != -1 && responseBufferedReader.ready());

responseBufferedReader is a BufferedReader wrapped around an InputStreamReader wrapped around the Socket's InputStream.
This code works most of the time, but it seems like checking for readChars != -1 and ready() are not reliable enough to indicate if we've read all of the content from the server. Comparing the number of read characters to the buffer size is also not reliable, since the server seems to be a little slow at sending the response back causing these numbers to differ.
I've tried changing the size of the character buffer; it helped, but it's still not working 100% of the time.
Is there a better and more reliable way to read entirely from a Socket without knowing the size of the expected response? I've been doing some research on SocketChannels, but I'm not sure if there's any benefit to be had by switching.
In case it helps, we're making a single, blocking Socket connection to the server. The Socket is configured for a 100 second timeout

Answer 1

You shouldn't be checking whether the BufferedReader is ready() to tell if you're done. It's possible that bytes are still being read off of the wire and the BufferedReader has nothing for you, but the Socket is not yet closed.

I'm not sure what value the BufferedReader is getting you (well, it might help your performance). I would expect the following to work better:

StringWriter response = new StringWriter();
PrintWriter writer = new PrintWriter(response);

char[] buf = new char[4096];
int readChars;
do {
   readChars = inputStreamReader.read(buf);
   writer.write(buf, 0, readChars);
} while(readChars != -1);

I think that you're probably dropping out of the loop when the network is the bottleneck - the loop processes the results fast enough so that the Reader isn't ready yet and you assume that you're done reading (even though you're not).

Answer 2

See the answers to the question at http://stackoverflow.com/questions/2717023/java-blocking-socket-returning-incomplete-bytebuffer

If the network breaks the message into packets, you will drop out of your loop before reading all packets.

Answer 3

See my answer to this question. The same answer applies whenever you are reading from a stream, and the only reliable way to know when you're done if EOF doesn't apply (as in this case) is to encode the end-of-message in the protocol somehow -- how you do it is up to you.

Edit:

If you can't modify the protocol to include message boundaries, you might investigate a timeout-based approach. You have to decide how long to wait before detecting the end of a message... but then you have to handle the case where you get two messages in the same stream. Can your code handle this situation now? If so, then your answer is to buffer the input and use that logic to detect record boundaries.