So I have a an application which is running on top of gridgain and does so quite successfully for about 12-24 hours of stress testing before it starts to act funny. After this period of time the application will suddenly start replying to all queries with the exception java.nio.channels.ClosedByInterruptException (full stack trace is at http://pastie.org/664717
The method that is failing from is (edited to use @stephenc feedback)
public static com.vlc.edge.FileChannel createChannel(final File file) {
FileChannel channel = null;
try {
channel = new FileInputStream(file).getChannel();
channel.position(0);
final com.vlc.edge.FileChannel fileChannel = new FileChannelImpl(channel);
channel = null;
return fileChannel;
} catch (FileNotFoundException e) {
throw new VlcRuntimeException("Failed to open file: " + file, e);
} catch (IOException e) {
throw new VlcRuntimeException(e);
} finally {
if (channel != null) {
try {
channel.close();
} catch (IOException e){
// noop
LOGGER.error("There was a problem closing the file: " + file);
}
}
}
}
and the calling function correctly closes the object
private void fillContactBuffer(final File signFile) {
contactBuffer = ByteBuffer.allocate((int) signFile.length());
final FileChannel channel = FileUtils.createChannel(signFile);
try {
channel.read(contactBuffer);
} finally {
channel.close();
}
contactBuffer.rewind();
}
The application basically serves as a distributed file parser so it does a lot of these types of operations (will typically open about 10 such channels per query per node). It seems that after a certain period it stops being able to open files and I'm at a loss to explain why this could be happening and would greatly appreciate any one who can tell me what could be causing this and how I could go about tracking it down and fixing it. If it is possibly related to file handle exhaustion, I'd love to hear any tips for finding out for sure... i.e. querying the JVM while it's running or using linux command line tools to find out more information about what handles are currently open.
update: I've been using command line tools to interrogate the output of lsof and haven't been able to see any evidence that file handles are being held open... each node in the grid has a very stable profile of openned files which I can see changing as the above code is executed... but it always returns to a stable number of open files.
Related to this question: http://stackoverflow.com/questions/1604935/freeing-java-file-handles