Shell script for testing

Question

I want a simple testing shell script that launches a program N times in parallel, and saves each different output to a different file. I have made a start that launches the program in parallel and saves the output, but how can I keep only the outputs that are different? Also how can I actually make the echo DONE! indicate the end?

#!/bin/bash

N=10

for((i=1; j<=$N; ++i)); do
    ./test > output-$N &
done
echo DONE!

Answer 1

In order to have your output indicate that all the processes are finished, you need to call wait:

#!/bin/bash
N=10

for((i=1; j<=$N; ++i)); do
    ./test > output-$N &
done
wait  #  wait until all jobs are finished
echo DONE!

Answer 2

You'll want to use the wait builtin.

wait [n ...]

Wait for each specified process and return its termination status. Each n may be a process ID or a job specification; if a job spec is given, all processes in that job’s pipeline are waited for. If n is not given, all currently active child pro- cesses are waited for, and the return status is zero. If n specifies a non-existent process or job, the return status is 127. Otherwise, the return status is the exit status of the last process or job waited for.

You could specify your jobs as %1, %2, ...:

wait %1 %2 %3 ...

but as long as you have no other child processes, you can just use it with no arguments; it'll then wait for all child processes to finish:

for ...; do
    ...
done
wait
echo "All done!"

Your separate question, how to keep only different outputs, is a little trickier. What exactly do you mean - different from what? If you have a baseline, you could do this:

for ...; do
    if diff -q $this_output $base_output; then
        # files are identical
        rm $this_output
     fi
done

If you want to keep all unique outputs, the algorithm's a little more complex, obviously, but you could still use diff -q to test for identical output.

Answer 3

With GNU Parallel http://www.gnu.org/software/parallel/ you can do:

/tmp/test > base; seq 1 10 | parallel -k "/tmp/test >output-{}; if diff -q output-{} base; then rm {}; fi" ; echo DONE

GNU Parallel is useful for other stuff. Watch the intro video to GNU Parallel: http://www.youtube.com/watch?v=OpaiGYxkSuQ