Detecting when a process has finished (but not exited)

Question

I have a program that's run in unix (that I have no control over) that when finished prints 'Completed successfully' but does not exit. I want to automatically detect when the process finishes (by checking the output of the command), so that I can kill the process and so that I can proceed do other activities. The complexity comes because I want to be able to run multiples of these scripts concurrently. (One activity I need to do requires the script to be called with various inputs, but each time the script runs it takes a while to return, and so I want to do them in parallel)

Has anyone done something similar to this? I could redirect the stderr and stdout output of the command to a temporary file which has a random file name, then tail the file and pipe to grep for the end conditions (I.e. the certain log lines). The problem is, surely tail -f would keep running, and so it would never exit. Should I poll? If so, what's the best approach?

Answer 1

You don't have to poll. You can use inotify tools, such as inotifywait, to detect when the output file has changed, and then grep it and kill the process if necessary. Happy hunting!

Answer 2

this might be overkill for bash, maybe? if you were using python, you could use the subprocess module keep a communication channel open between your own program and your child scripts. even better, you could use the threading module to start multiple runs of the child script in parallel.

does the script output anything AFTER it outputs "completed successfully" ? if not, you could just poll for the last x characters it has outputed (to a random file like you suggested, maybe using tail -n 1 to get the last line) and see if you just saw 'completed successfully'. but if the script keeps printing stuff afterwards, you might miss the line.

or you could grep for "completed successfully" in the output.

Answer 3

A few random ideas. (1) If it's your program, you could modify it to indicate doneness more usefully. (2) Looking at the process list ( ps ax ) you should be able to see it go into an I/O wait when it tries to output the "Completed" message. (3) I believe pseudo tty's exist for this purpose, that you can watch their stdout directly without having to muck about with temporary files.

Answer 4

Hi,
1st possible solution:
your_program > tempfile1
grep -i "Completed successfully" tempfile1; if [ $? -eq 0 ] ; then i=`pidof your_program`; kill $i; fi

First is redirection. Second checks for the exit status of grep. If it is 0 (success) it gets the pid of your program and kills it.
This was a "think-fast" exercise. It should work. A drawback is that you can only run one instance of your program at a time, but of course an adaptation can be made and you could run it in parallel.

edit-27.05.2010:
2nd possible solution:
tail --pid=`pidof your_program` -f -n0 tempfile1 | awk '/Completed/ {if ($1=="Completed") {system("i=`pidof your_program`; kill $i")}}' &

Well, with this long line you don't need a loop/crontab anymore. Explanation:
--pid=`pidof your_program` makes tail die after the specified PID is ended. This way, when the program gets killed, tail dies along with it.
-n0 ignores other "Completed" statements in tempfile1 (so you don't have to worry to always truncate tempfile1 first).
awk searches for the "Completed" statement thrown out by your program and the {system} part executes the kill on your program.
Also don't forget the & at the end as it backgrounds the entire thing.
Have fun.

Answer 5

Here's my stab at it, it works pretty well, but is rather spammy because each time kill is called a killed message echoing the entire subshell script is output to console and I can't seemt to supress it. I will update if I find a way to get rid of the spammyness, (by the way, redirecting stdout and stderr to /dev/null does not help.)

Note: You need bash4 to run this; $BASHPID is only available from 4 onwards, and $$ is not an alternative because it gives the pid of the parent shell, not the subshell.

!/bin/bash

for i in {1..8}; do
  (
    mypid=$BASHPID
    (
      #subshell to simulate started process
      sleep 2
      echo finished
      while : ; do
        :
      done
    ) | \
    {
      while read line; do
        if [[ $line == finished ]]; then
          break;
        fi
      done
      echo process done, killing $mypid...
      kill -9 $mypid
    }
  ) &
done
wait

Answer 6

The following wrapper script will call your real command, piping output to tee which will write to a fifo. The wrapper script will kill your real command, when the expected string is greped from the fifo:

#!/bin/bash

# cmd to run
expected_output=$1
shift
cmd=("$@")
# where to read commands output
mkfifo /tmp/killonoutput.$$
# start cmd async 
"${cmd[@]}"|tee /tmp/killonoutput.$$ 2>/dev/null &

if grep -q --max-count=1 "$expected_output" /tmp/killonoutput.$$;then
    kill %1 
    echo "Killed ${cmd[@]}" 
fi
# grep returned so fifo was closed
rm  /tmp/killonoutput.$$

Sample execution:

./killonoutput.sh "Finished" bash -c "echo sleeping;sleep 3;echo Finished; sleep 10000" 
sleeping
Finished
Killed bash -c echo sleeping;sleep 3;echo Finished; sleep 10000
./killonoutput.sh: line 17: 19553 Terminated              "${cmd[@]}"
     19554                       | tee /tmp/killonoutput.$$ 2> /dev/null

Answer 7

#!/usr/bin/expect

spawn /home/foo/myjob
expect "Completed successfully"