Calling a shell script from java hangs

Question

So I'm trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file. This hangs the process and never completes.

However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.

Is it because of amount of data that the Java program never completes? If so, is there any documentation as such? or is there any limit on the amount of data(documented)?

Here's how you can simulate this scenario.

Java file will look like:

import java.io.InputStream;

public class LotOfOutput {

    public static void main(String[] args) {
        String cmd = "sh a-script-which-outputs-huuggee-data.sh"; 
        try {
            ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
            pb.redirectErrorStream(true);
            Process shell = pb.start();
            InputStream shellIn = shell.getInputStream();
            int shellExitStatus = shell.waitFor();
            System.out.println(shellExitStatus);
            shellIn.close();
        } catch (Exception ignoreMe) {
        }
    }
}

The script 'a-script-which-outputs-huuggee-data.sh' may look like:

#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1

count=1
while [ $count -le 1000 ]
do
        cat some-big-file
        ((count++))
done

echo
echo Yes I m done

Free beer for the right answer. :)

Answer 1

It's because you're not reading from the Process' output.

As per the class' Javadocs, if you don't do this then you may end up with a deadlock; the process fills its IO buffer and waits for the "shell" (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.

You'll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.

Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won't - both informative articles about common problems with Process.

Answer 2

You're never reading the input stream, so it's probably blocking because the input buffer is full.

Answer 3

The input/output buffer have a limited size (depending on the operating system). If I remember correctly this wasn't big or Windows XP at least. Try creating a thread that reads the InputStream as fast as possible.

Something along these lines:

class StdInWorker
            implements Worker
    {
        private BufferedReader br;
        private boolean run = true;
        private int linesRead = 0;

        private StdInWorker (Process prcs)
        {
            this.br = new BufferedReader(
                    new InputStreamReader(prcs.getInputStream()));
        }

        public synchronized void run ()
        {
            String in;
            try {
                while (this.run) {
                    while ((in = this.br.readLine()) != null) {
                        this.buffer.add(in);
                        linesRead++;
                    }

                    Thread.sleep(50);
                }
            }
            catch (IOException ioe) {
                ioe.printStackTrace();
            }
            catch (InterruptedException ie) {}
        }
    }
}