Java hangs even though script's execution is completed

Question

I'm trying to execute a script from within my java code which looks like:

Process p = Runtime.getRuntime().exec(cmdarray, envp, dir); // cmdarray is a String array
// consisting details of the script and its arguments

final Thread err = new Thread(...); // Start reading error stream
err.start();
final Thread out = new Thread(...); // Start reading output stream
out.start();
p.waitFor();
// Close resources 

The execution of the script is over(it's pid is no more), but java is stuck on waitFor() method of the process!. And yes, I'm reading output and error streams in 2 separate threads. Yes, they are being joined at the end(after waitFor()).

The script basically installs a few RPMs(like 10 or so) and configures them. So the script runs for a little over 60 seconds.

It looks similar to the following:

#!/bin/sh

#exec 3>&1 >/var/log/some_log 2>&1

# If the above line is uncommented, Java recognizes that the 
# process is over and terminates fine.

tar xzf a-package-having-rpms.tar.gz
cd unpacked-folder
(sh installer-script.sh) #This installs 10 odd rpms in a subshell and configures them
cd ..
rm -rf unpacked-folder

exit 0

Shockingly enough, if I put the following line in the script(at the top), Java understands the script is over and it terminates the process perfectly.

exec 3>&1 > /var/log/some_log 2>&1

For the record, the script doesn't generate any output. Zero chars!. So putting exec statement there makes no sense!

But still, magically enough, putting exec statement in the script makes java work!. Why??

How can I avoid that illogical exec statement in the script?.

If you are interested in what installer-script.sh looks like then:

#!/bin/sh

exec 3>&1 >>/var/log/another-log.log 2>&1
INSDIR=$PWD
RPMSDIR=$INSDIR/RPMS
cd $RPMSDIR
#
rpm -i java-3.7.5-1.x86_64.rpm
rpm -i --force perl-3.7.5-1.x86_64.rpm
rpm -i --nodeps mysql-libs-5.0.51a-1.vs.i386.rpm
rpm -i --nodeps mysql-5.0.51a-1.vs.i386.rpm
rpm -i --nodeps mysql-server-5.0.51a-1.vs.i386.rpm
rpm -i --nodeps perl-DBD-MySQL-3.0007-2.el5.i386.rpm
rpm -i --nodeps perl-XML-Parser-2.34-6.1.2.2.1.i386.rpm
.
.
.

Now, why exec command in the first script is required for Java to know that the process is over? How can I avoid that exec?, esp. since the first script doesn't produce any output.

Waiting for answers with bated breath!

Answer 1

If you're running on Linux/Unix, try not consuming stdout/stderr.

Last time I had to run an external script from Java I ended up having to check for the OS and attach output eater threads only if on Windows.

Answer 2

I would also pump standard input as well. It's a good idea to pump all three standard streams. I can't explain it, but I've experienced similar problems before with processes hanging, and it was the lack of a stdin pump that was the problem.

Answer 3

I haven't tried in Linux, but I ran into this problem in Windows. I think I had to close the process's standard in stream before it would terminate.

Answer 4

You just can't exec a script from Java and wait for it. You must execute the interpreter of that script. The cmdarray should start with "bash" and not "myscript.bash".

Answer 5

A likely cause is that installing the packages starts background processes that keep one or both of your pipes (stdout/stderr) open. Therefore your threads will not terminate. This means that these background processes do not daemonize properly. A proper daemon replaces its original stdin/stdout/stderr with /dev/null or a log file after it has finished initializing.

Answer 6

My guess is that Java doesn't think the script is over until the pipes it passed to it via stdin/stdout/stderr are closed by the sub-process. That is, there are no more active reader processes on stdin, no more active writer processes on stdout/stderr.

When you're reading on a pipe, you don't receive an end-of-file indication until there are no more processes that have the pipe open for output. So if a process forks and the new process inherits an open file handle, then the original process terminates, there's still a process with the file open, and a reader will still wait.

Similarly with a pipe you're writing, you won't receive a "broken pipe" signal until the last reader closes the pipe.

This problem generally arises when your script forks off background tasks (like newly-installed services) which inherit stdin/stdout/stderr.

By using exec, you're explicitly breaking the inheritance chain of these pipes so that the background processes don't use them.

If on Linux, check /proc/*/fd for any new services and see if their stdin/stdout/stderr is the same pipe that your java process passes to your script.

The same situation often happens when you run the /etc/init.d/xxx scripts: they complete normally when you run them from the command-line but seem to hang when you run them from some kind of monitor.

EDIT:

You say that the installer script contains the line:

exec 3>&1 >>/var/log/another-log.log 2>&1

The first term, 3>&1, clones stdout to file-descriptor 3 (see Redirections in man bash). As far as I know, fd 3 has no special meaning. Then it replaces stdout by opening /var/log/another-log.log and replaces stderr by cloning stdout. See the Redirections section of the bash man page

This means the the new file-descriptor 3 is open for writing on the pipe that was originally passed in as STDOUT. Programs that expect to be system service daemons will often close file descriptors 0 (STDIN), 1 (STDOUT) and 2 (STDERR) but may not bother with any others. Also, now that the shell has opened FD-3 it will pass that open file to any command it executes, including background commands.

Do you know if there's any particular reason that the installer opens FD 3? My guess is that if you simply remove the "3>&1" term from the installer your problem will be solved. This will allow you to remove the exec from your script entirely

Answer 7

Hey, pavanlimo -

It sounds like you're understanding bits and pieces of the problem, but you're still not quite getting the "Big Picture".

1. Yes, it's possible to call a shell script from a Java program. Easy, in fact! <= You already know this.

2. Yes, generally, you should be able to see anything that's printed to stderr or stdout.

3. And yes, you should be able to redirect stderr and/or stdout to a file.

4. Your problem is evidently that you're not seeing WHAT you expect to see, WHEN you expect to see it. This is normal, this is expected ... and, in the bigger scheme of things, this is HIGHLY DESIRABLE.

The central issue, I believe, is "I/O buffering".

Buffering is a Good Thing.

I strongly encourage you to: 1. Create a simple, single-threaded command line Java test program that calls a shell script. <= You should see "expected results"

1. Elaborate the test program with threads. Depending on how the program is structured, you may or may not get "expected results".

2. Substitute a C program (or another Java program) for the shell script <= This makes it easier to use "stderr", which is UNBUFFERED, vs "stdout" You should also lose the "exec" nonsense - that was a red herring!

3. Experiment, and see what you learn.

Good luck - and please DON'T just jump to the first conclusion that seems half-way plausible. Always TEST your theories.

PS: And repeat my mantra: "I/O buffering is Good" ;-)

PPS:

Q: This question has been viewed only 38 times so far!

A: "You must learn patience, young Grasshopper" ;-)

Q: This is kinda critical for me!

A: "Humility must you learn, too" ;-)