Efficient bulk data transfer through I/O in Java

Question

Is there an efficient mechanism in Java for transferring large amounts of data (~10 GB) back and forth between the disk and the network on a low-end machine? Low-end machine meaning a machine with much less than 10 GB of main memory. I want to efficiently transfer ~10GB portions of a file from one machine to another.

Answer 1

The most efficient means of I/O in Java is the NIO libraries. If your data is appropriate applying a compression stream filter would help as well.

Answer 2

traditional socket and file io would be fine.

in java 6, there is no direct file-socket io optimization as far as I know.

even if there is, as rumored in java 7, it won't improve performance very much. your code would be either disk bound or network bound. mem copy shouldn't matter.

Answer 3

I think you need to just copy an InputStream to an OutputStream. I'd start off with using somebody else's implementation commons-io:IOUtils.copyLarge(InputStream,OutputStream)

InputStream is = null;
OutputStream os = null;
try{
    InputStream is = new FileInputStream("yourfile.bin");
    OutputStream os = remoteSocket.getOutputStream();
    IOUtils.copyLarge(is,os);
}finally{
    IOUtils.closeQuietly(is);
    IOUtils.closeQuietly(out);
}

This should be a good starter for ten. If you need higher through put, you can start my putting the read and write actions in separate threads, this in theory should ensure you completely saturate the slowest of the links, but it may be a better choice to use the FileChannel#transferTo method if using traditional stuff just isn't enough.

Answer 4

FYI, transferTo() only is only optimized between files, not file-socket, in java 6