Process Builder waitFor() issue and Open file limitations

Question

I have inherited some code:

Process p = new ProcessBuilder("/bin/chmod", "777", path).start();
p.waitFor();

Basically, there is for some ancient and highly voodoo based reason for storing key/value pairs on disk as files. I don't really want to go into it.

However, I am left with a bunch of IO exceptions: Exception :Cannot run program "/bin/chmod": java.io.IOException: error=24, Too many open files Message: Cannot run program "/bin/chmod": java.io.IOException: error=24, Too many open files

And by a bunch I mean in the realms of 10k - millions

I get the feeling the waitFor call was to stop these from occurring waiting for the process to complete it and exit back, however I think the chmod is returning a result before the file is actually closed. Does anyone know if that would be the cause of these exceptions?

My other inclination is that the opening and closing of thousands of files is not happening quickly enough on the java end and that there is something else going on, maybe something like that there is some form of file buffer that isn't getting cleared out when fw.close() is being called.

I am pretty new to java and this was a hell weird one that has me stumped. (gladly the app still runs somehow.. after spitting out a very large log file that is)

Can anyone else think of a way to get around this, clearing buffers or increasing the files open limit to something where the jvm can keep up with itself (assuming that is the problem)

Answer 1

It seems unlikely that the process would actually complete without closing the files. Could this be happening in a very large # of threads? Or perhaps some of them are not actually completing (ie, it is hanging at waitFor in some cases)?

Otherwise, I think you will be stuck with increasing the open files limit. Assuming that this is a Unix-like system, the "ulimit" command is probably what you are looking for.

Answer 2

If you're using JAVA 6, you could also try the new setters (for read,write,execute) on the File object. Might be slower, but it should work.

Answer 3

I presume you are running these chmod commands in a loop - otherwise I don't see why you'd get so many exceptions. It's possible that you're hitting a deadlock because you're not reading the output of the spawned processes. That certainly used to bite me back in the pre-ProcessBuilder, Runtime.exec() days.

Change your code snippet to the above pattern:

try {
    ProcessBuilder pb = new ProcessBuilder("/bin/chmod", "777", path);    
    pb.redirectErrorStream(true); // merge stdout, stderr of process

    Process p = pb.start();
    InputStreamReader isr = new  InputStreamReader(p.getInputStream());
    BufferedReader br = new BufferedReader(isr);

    String lineRead;
    while ((lineRead = br.readLine()) != null) {
        // swallow the line, or print it out - System.out.println(lineRead);
    }

    int rc = p.waitFor();
    // TODO error handling for non-zero rc
}
catch (IOException e) {
    e.printStackTrace(); // or log it, or otherwise handle it
}
catch (InterruptedException ie) {
    ie.printStackTrace(); // or log it, or otherwise handle it
}

(credit: this site) and see if that helps the situation.

Answer 4

Thanks for the help guys, this should sort out a load of weirdness going on elsewhere because of it.

Using your(Vinay) example and the stream closings:

try{ 
  fw.close();

  ProcessBuilder pb = new ProcessBuilder("/bin/chmod", "777", path);

  pb.redirectErrorStream(true); // merge stdout, stderr of process
  p = pb.start();

  InputStreamReader isr = new  InputStreamReader(p.getInputStream());
  BufferedReader br = new BufferedReader(isr);

  String lineRead;
  while ((lineRead = br.readLine()) != null) {
    // swallow the line, or print it out - System.out.println(lineRead);
  }

} catch (Exception ioe) {
  Logger.logException(Logger.WARN, ioe.getMessage(), ioe);
} finally {
  try {
    p.waitFor();//here as there is some snipped code that was causing a different
                // exception which stopped it from getting processed

    //missing these was causing the mass amounts of open 'files'
    p.getInputStream().close();
    p.getOutputStream().close();
    p.getErrorStream().close(); 

  } catch (Exception ioe) {
    Logger.logException(Logger.WARN, ioe.getMessage(), ioe);
  }
}

Got the idea from John B Mathews post.