Socket closing problem - last portion of data is lost.

Question

I have a Java program that accepts connections, receives HTTP requests and sends HTTP replies and some data stored in file (this is a part of caching proxy). Removing everything irrelevant, my code looks like this:

FileInputStream fileInputStream = new FileInputStream(file);
OutputStream outputStream = socket.getOutputStream();
byte[] buf = new byte[BUFFER_SIZE];
int len = 0;
while ((len = fileInputStream.read(buf)) > 0) {
    outputStream.write(buf, 0, len);
}
outputStream.flush();
socket.close();

This code is executed in particular thread for each connected client.

When I deal with small files (.htm, .gif, .swf, etc.), everything works fine (however, I don't see anything wrong in browser). But when I download large files (.iso), especially several files simultaneously, when system is under load, sometimes I get really strange behavior. Browser downloads 99.99% of a file and when there are less than BUFFER_SIZE of undownloaded bytes, downloading stops for a few seconds and then browser says that error has occured. I can not understand what happens, because all data is successfully read and even all data is successfully written to outputStream. As you can see, I even do flush(), but it takes no result.

Can anyone explain me what happens?

EDIT
Uploaded project to filehosting.org.
Download source files. There is zip archive with source code, Build.xml and Readme.txt. Use ant to build solution. Described problem occurs in ClientManager.java, you'll find a comment there.

Answer 1

Not sure why that would happen, I don't see a problem with your code. Have you tried using a BufferedInputStream to enclose the FileInputStream?

Answer 2

I think the most probable cause is that you closed the connection without waiting for the data to be fully transmitted.

You can ask Java to wait a little while for you,

socket.setOption(SocketOptions.SO_LINGER, new Integer(60));

Answer 3

Based on a quick trawl through the JDK 1.6 codebase:

• socket.getOutputStream() returns a SocketOutputStream instance
• flush() indeed has no effect on a SocketOutputStream instance
• write() on a SocketOutputStream instance does not seem to buffer anything in Java code
• shutdownOutput() should ensure that any outstanding data is written before shutting down the output side of the socket. At least, the comments say that.

However, some of the Socket etc implementation is native methods, and I didn't delve into that.

Based on what I could tell, the "correct" sequence would be:

socket.shutdownOutput();
socket.close();

However, you say you've tried that. Is it possible that the application at the other end is closing the TCP/IP connection early?

Another thought: you tried setSoLinger(true, 60000), but 60000 seconds is possibly longer that the OS allows. Try setSoLinger(true, 60), and try doing it before you open the output stream.

Answer 4

I am used to sending the length first, and then waiting until I get that data, but if you can't do that, this blog should be helpful:

http://vadmyst.blogspot.com/2008/04/proper-way-to-close-tcp-socket.html

This is for .NET, but the basic logic should be the same.

Once you have sent the data, you will need to have the sending side close their connection, then do shutdownOutput and have the other side continue to read until there is a situation where nothing can be read, or an exception is thrown, and then you should have everything.