Java NIO: Sending large messages quickly leads to truncated packets and data loss

Question

I've got this nasty problem where sending multiple, large messages in quick succession from a Java (NIO) server (running Linux) to a client will lead to truncated packets. The messages have to be large and sent very rapidly for the problem to occur. Here's basically what my code is doing (not actual code, but more-or-less what's happening):

//-- setup stuff: --
Charset charset = Charset.forName("UTF-8");
CharsetEncoder encoder = charset.newEncoder();
String msg = "A very long message (let's say 20KB)...";

//-- inside loop to handle incoming connections: --
ServerSocketChannel ssc = (ServerSocketChannel)key.channel();
SocketChannel sc = ssc.accept();
sc.configureBlocking(false);
sc.socket().setTcpNoDelay(true);
sc.socket().setSendBufferSize(1024*1024);

//-- later, actual sending of messages: --
for (int n=0; n<20; n++){
  ByteBuffer bb = encoder.encode(CharBuffer.wrap(msg+'\0'));
  sc.write(bb);
  bb.rewind();
}

So, if the packets are long enough and sent as quickly as possible (i.e. in a loop like this with no delay), then on the other end it often comes out something like this:

[COMPLETE PACKET 1]
[COMPLETE PACKET 2]
[COMPLETE PACKET 3]
[START OF PACKET 4][SOME OR ALL OF PACKET 5]

There is data loss, and the packets start to run together, such that the start of packet 5 (in this example) arrives in the same message as the start of packet 4. It's not just truncating, its running the messages together.

I imagine that this is related to the TCP buffer or "window size", or that the server here is just providing data faster than the OS, or network adapter, or something, can handle it. But how do I check for, and prevent it from happening? If I reduce the length of message per use of sc.write(), but then increase the repetitions, I'll still run into the same problem. It seems to simply be an issue with the amount of data in a short amount of time. I don't see that sc.write() is throwing any exceptions either (I know that in my example above I'm not checking, but have in my tests).

I'd be happy if I could programmatically check if it is not ready for more data yet, and put in a delay, and wait until it is ready. I'm also not sure if "sc.socket().setSendBufferSize(1024*1024);" has any effect, or if I'd need to adjust this on the Linux side of things. Is there a way to really "flush" out a SocketChannel? As a lame workaround, I could try to explicitly force a complete send of anything that is buffered any time I'm trying to send a message of over 10KB, for example (which is not that often in my application). But I don't know of any way to force a send of the buffer (or wait until it has sent). Thanks for any help!

Answer 1

I haven't done any NIO programming, but according to the Javadocs, sc.write() will not write the entire ByteBuffer if the SocketChannel is in non-blocking mode (as yours is) and the socket's output buffer is full.

Because you are writing so quickly, it is very likely that you are flooding your connection and your network or receiver cannot keep up.

I'd be happy if I could programmatically check if it is not ready for more data yet

You need to check the return value of sc.write() to find out whether your output buffer is full.

Answer 2

You are not checking the return value of:

sc.write(bb);

This returns the number of bytes written, which might be less than the data available in your buffer. Because of how nio works you can probably just call remaining() on your bytebuffer to see if there are any left.

Answer 3

There are many reasons why sc.write() would not send some or all of the data. You have to check the return value and/or the number of bytes remaining in the buffer.

for (int n=0; n<20; n++){
  ByteBuffer bb = encoder.encode(CharBuffer.wrap(msg+'\0'));
  if(sc.write(bb) > 0 && bb.remaining() == 0) {
    // all data sent
  } else {
    // could not send all data.
  }
  bb.rewind();
}

Answer 4

Don't assume you have any control over what data ends up in which packet.

More here: http://stackoverflow.com/questions/453609/whats-the-best-way-to-monitor-a-socket-for-new-data-and-then-process-that-data/453951#453951

Answer 5

You are using the non-blocking mode: sc.configureBlocking(false);

Set blocking to true and your code should work as it is. Suggestions made by others here to check send count and loop will also work.