How to make a pipe loop in bash

Question

Assume that I have programs P0, P1, ...P(n-1) for some n > 0. How can I easily redirect the output of program Pi to program P(i+1 mod n) for all i (0 <= i < n)?

For example, let's say I have a program square, which repeatedly reads a number and than prints the square of that number, and a program calc, which sometimes prints a number after which it expects to be able to read the square of it. How do I connect these programs such that whenever calc prints a number, square squares it returns it to calc?

Edit: I should probably clarify what I mean with "easily". The named pipe/fifo solution is one that indeed works (and I have used in the past), but it actually requires quite a bit of work to do properly if you compare it with using a bash pipe. (You need to get a not yet existing filename, make a pipe with that name, run the "pipe loop", clean up the named pipe.) Imagine you could no longer write prog1 | prog2 and would always have to use named pipes to connect programs.

I'm looking for something that is almost as easy as writing a "normal" pipe. For instance something like { prog1 | prog2 } >&0 would be great.

Answer 1

This is a very interesting question. I (vaguely) remember an assignment very similar in college 17 years ago. We had to create an array of pipes, where our code would get filehandles for the input/output of each pipe. Then the code would fork and close the unused filehandles.

I'm thinking you could do something similar with named pipes in bash. Use mknod or mkfifo to create a set of pipes with unique names you can reference then fork your program.

Answer 2

I second mharen's response: use a 'for .. in' loop with the seq command (to go from 0 to n-1)

Answer 3

@mharen (40261) and PW (40343):

I'd look into just using a loop for this. Piping isn't really designed for this sort of thing but bash scripting can easily handle it with a loop.

use a 'for .. in' loop with the seq command (to go from 0 to n-1)

I'm afraid that I haven't been clear enough in my question as I do not see anyway to solve this using a loop. (It could also be I'm completely missing the obvious solution, of course.)

The idea is that all processes run simultaneously and that during their execution their output is redirected to the input of the next process in the "loop" (i.e. the pipe loop, not a while/for loop). So, sticking to the example, I would like a way to write both calc | square and square | calc at the same time (for the same instances of calc and square, of course).

@Dr. UNIX (40339):

I'm thinking you could do something similar with named pipes in bash. Use mknod or mkfifo to create a set of pipes with unique names you can reference then fork your program.

In the past I have indeed used some fifos to (sort of) solve the problem. However, I cannot say I have found it to be an easy and satisfactory way. I would love to have a solution that is about as easy as just writing a pipe.

Answer 4

I doubt sh/bash can do it. ZSH would be a better bet, with its MULTIOS and coproc features.

Answer 5

I wrote the following to try to fulfill your example--let us know how it suits you (I'm using bash, but I think it will work as well in at least in ksh):

#!/bin/sh

function square
{
    while read n; do
        echo "$n * $n" | bc
    done
}

function calc
{
    i=0
    while [ $i -lt 10 ]; do
        echo $i
        i=$((i + 1))
    done | square
}

calc

Answer 6

Named pipes.

Create a series of fifos, using mkfifo

i.e fifo0, fifo1

Then attach each process in term to the pipes you want:

processn < fifo(n-1) > fifon

Answer 7

A named pipe might do it:

$ mkfifo outside
$ <outside calc | square >outside &
$ echo "1" >outside ## Trigger the loop to start

Answer 8

After spending quite some time yesterday trying to redirect stdout to stdin, I ended up with the following method. It isn't really nice, but I think I prefer it over the named pipe/fifo solution.

read | { P0 | ... | P(n-1); } >/dev/fd/0

The { ... } >/dev/fd/0 is to redirect stdout to stdin for the pipe sequence as a whole (i.e. it redirects the output of P(n-1) to the input of P0). Using >&0 or something similar does not work; this is probably because bash assumes 0 is read-only while it doesn't mind writing to /dev/fd/0.

The initial read-pipe is necessary because without it both the input and output file descriptor are the same pts device (at least on my system) and the redirect has no effect. (The pts device doesn't work as a pipe; writing to it puts things on your screen.) By making the input of the { ... } a normal pipe, the redirect has the desired effect.

To illustrate with my calc/square example:

function calc() {
  # calculate sum of squares of numbers 0,..,10

  sum=0
  for ((i=0; i<10; i++)); do
    echo $i                   # "request" the square of i

    read ii                   # read the square of i
    echo "got $ii" >&2          # debug message

    let sum=$sum+$ii
  done

  echo "sum $sum" >&2           # output result to stderr
}

function square() {
  # square numbers

  read j                         # receive first "request"
  while [ "$j" != "" ]; do
    let jj=$j*$j
    echo "square($j) = $jj" >&2  # debug message

    echo $jj                     # send square

    read j                       # receive next "request"
  done
}

read | { calc | square; } >/dev/fd/0

Running the above code gives the following output:

square(0) = 0
got 0
square(1) = 1
got 1
square(2) = 4
got 4
square(3) = 9
got 9
square(4) = 16
got 16
square(5) = 25
got 25
square(6) = 36
got 36
square(7) = 49
got 49
square(8) = 64
got 64
square(9) = 81
got 81
sum 285

Of course, this method is quite a bit of a hack. Especially the read part has an undesired side-effect: termination of the "real" pipe loop does not lead to termination of the whole. I couldn't think of anything better than read as it seems that you can only determine that the pipe loop has terminated by try to writing write something to it.

Answer 9

A command stack can be composed as string from an array of arbitrary commands and evaluated with eval. The following example gives the result 65536.

function square ()
{
  read n
  echo $((n*n))
}    # ----------  end of function square  ----------

declare -a  commands=( 'echo 4' 'square' 'square' 'square' )

#-------------------------------------------------------------------------------
#   build the command stack using pipes
#-------------------------------------------------------------------------------
declare     stack=${commands[0]}

for (( COUNTER=1; COUNTER<${#commands[@]}; COUNTER++ )); do
  stack="${stack} | ${commands[${COUNTER}]}"
done

#-------------------------------------------------------------------------------
#   run the command stack
#-------------------------------------------------------------------------------
eval "$stack"