Process.waitFor(), threads, and InputStreams

Question

In pseudocode, here's what I'm doing:

Process proc = runtime.exec(command);
processOutputStreamInThread(proc.getInputStream());
processOutputStreamInThread(proc.getErrorStream());
proc.waitFor()

However, sometimes processOutputStreamInThread doesn't see any output and sometimes it does. Roughly, the method creates a BufferedInputStream of the command's output and sends it to a logger.

Based on what I'm seeing, I'm guessing that command need not have all it's output dumped into the streams fed by getInputStream() and getErrorStream(), thus allowing the stream to be empty.

The results of my trials are the following questions:

(1) Does waitFor() in java.lang.Process require the executed program's output to have been read before it returns?

The documentation only states:

causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.

(2) Under what conditions do the streams provided by getInputStream and getErrorStream need to be closed and/or are they closed automatically?

The documentation only states:

Gets the error stream of the subprocess. The stream obtains data piped from the error output stream of the process represented by this Process object.

Implementation note: It is a good idea for the input stream to be buffered.

One user reports that he had to close the streams himself, but I get an exception at least part of the time indicating that the stream is already closed when I attempt to do so.

Edit: changed getOutputStream to getInputStream, now present above.

Resolution: The problem ended up being that in certain cases the threads used to process the output stream wouldn't run until after my very short-lived process had completed, resulting in the input stream giving me no data. waitFor didn't do any waiting for the output of the executed program. Rather, the program ran and terminated before any output could be gathered.

I used threads because I'm not sure how much output I was going to get on standard error and standard output and I wanted to be able to process both simultaneously, without blocking one or the other should only one of them have data available. But, because my threads can't consistently read the executed program's output, it's a non-solution.

My final coded looked something like this:

ProcessBuilder pb = new ProcessBuilder(cmdargs);
pb.redirectErrorStream(true);
Process proc = pb.start();
processOutputStream(proc.getInputStream());
proc.waitFor()

Answer 1

I think this is slightly counter-intuitive, but:

getOutputStream Gets the output stream of the subprocess. Output to the stream is piped into the standard input stream of the process represented by this Process object. Implementation note: It is a good idea for the output stream to be buffered. Returns: the output stream connected to the normal input of the subprocess.

I read that as this Output stream from the main process and is attached to the standard input of the sub-process, so when you write to getOutputStream().write() you are actually writing on the stdin.

Do you possibly want to use .getInputStream()?

Returns: the input stream connected to the normal output of the subprocess.

As for Process.Waitfor() the API docs say:

causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.

I would say the thread this is called in will be blocked until such a time as the process has finished executing - you may still be processing output at this stage depending on the other threads or they may have finished before your thread returns.

Answer 2

If your external process expects something on its stdin, you MUST close the getOutputStream. Otherwise you will waitFor forever.

Here is the When Runtime.exec() won't article from JavaWorld which describes different pitfalls of exec method and how to avoid them.

From my experience it's better to consume STDOUT and STDERR of child process ( until they EOF ) and then block in waitFor. Hopefully at this point you won't have to waitFor long.

An answer to Kaleb's question. Under normal conditions you shouldn't close the streams, however because you are waitingFor and for whatever reason it does not have a timeout, you may need to close these streams if you encounter some error conditions in the output and don't want to process child's output further. However, whether the child program will terminate (crash) when it's STDOUT or STDERR pipe is closed at the other end is totally up to that child's implementation. Most shell programs though, will terminate under such conditions.

I really wish waitFor had some meaningful timeout though and the Process had a documented way of cleaning up its resources when you've decided to abandon its monitoring.