Java - Strange hanging socket?

Question

I have a client-server application with a java server. It works pretty much perfect, except that after an extended period of time, all of a sudden, a socket keeps hanging. This socket is but one of many, the rest seem to do fine still, but once it gets to the socket, the server simply does not go past the sending line. These are the relevant pieces of the code:

Socket socket; // A normal socket
out  = new PrintWriter(socket.getOutputStream(), true); // The outstream
out.println(msg + "\0"); // This command is used to send stuff, msg is a String

No exceptions are thrown, the line application simply does not seem to get past the line:

out.println(msg + "\0");

I do know that the String is a good one, cause 4 or 5 other sockets before this one could send it just fine. Also note that, as far as I know, this socket could send hundreds of messages just fine before it suddenly hangs. Does anyone have a clue what kind of error I should be looking for?

Answer 1

There are many things that could cause this, but it sounds like you're probably getting backpressured at the TCP level.

Ordinarily, when you send data on a socket, it's simply buffered and the send call (in your case, println() and flush()) can return before the data is actually sent to the network. It could be that all your previous writes on this socket were just buffered to the local SO_SNDBUF and you just filled it up.

(1) Can you trace this using tcpdump or Wireshark and find the exact connection that's hanging (I know this gets hard with many connections)? Examining the TCP window size at the time your program hangs will tell you more about whether you're blocking in Java itself, or at the network level.

(2) Dump the stack and post it here. Java Sockets have an internal lock to prevent two threads from writing to them at once. Are you certain only one thread is accessing that socket at the moment you try writing?

You definitely want to try doing a

$ jstack <PID>

when it's frozen to see exactly where it's hanging, and if the writing thread is attempting to get any locks.

Answer 2

The Socket object's OutputStream will block on a write under the same conditions that the underlying send() call will block: when the local socket buffers are full. That it remains blocked indicates that the output buffer cannot be emptied, which in turn indicates that the read buffer on the other end of the connection is full, and that the application handling the other end of the connection is not reading data that has already arrived.