Forking / Multi-Threaded Processes | Bash

Question

Hi all -

I would like to make a section of my code more efficient. I'm thinking of making it fork off into multiple processes and have them execute 50/100 times at once, instead of just once.

For example (pseudo):

for line in file; do foo; foo2; foo3; done

I would like this for loop to run multiple times. I know this can be done with forking. Would it look something like this?

while(x <= 50) parent(child pid) { fork child() } child { do foo; foo2; foo3; done return child_pid() }

Or am I thinking about this the wrong way?

Thanks!

Answer 1

Let me try example

for x in 1 2 3 ; do { echo a $x ; sleep 1 ; echo b $x ; } &  done ; sleep 10

And use jobs to see what's running.

Answer 2

I don't know of any explicit fork call in bash. What you probably want to do is append & to a command that you want to run in the background. You can also use & on functions that you define within a bash script:

do_something_with_line()
{
  line=$1
  foo
  foo2
  foo3
}

for line in file
do
  do_something_with_line $line &
done

EDIT: to put a limit on the number of simultaneous background processes, you could try something like this:

for line in file
do
  while [`jobs | wc -l` -ge 50 ]
  do
    sleep 5
  done
  do_something_with_line $line &
done

Answer 3

In bash scripts (non-interactive) by default JOB CONTROL is disabled so you can't do the the commands: job, fg, and bg.

Here is what works well for me:

#!/bin/sh

set -m # Enable Job Control

for i in `seq 30`; do # start 30 jobs in parallel
  sleep 3 &
done

# Wait for all parallel jobs to finish
while [ 1 ]; do fg 2> /dev/null; [ $? == 1 ] && break; done