Synchronize shell script execution

Question

A modified version of a shell script converts an audio file from FLAC to MP3 format. The computer has a quad-core CPU. The script is run using:

./flac2mp3.sh $(find flac -type f)

This converts the FLAC files in the flac directory (no spaces in file names) to MP3 files in the mp3 directory (at the same level as flac). If the destination MP3 file already exists, the script skips the file.

The problem is that sometimes two instances of the script check for the existence of the same MP3 file at nearly the same time, resulting in mangled MP3 files.

How would you run the script multiple times (i.e., once per core), without having to specify a different file set on each command-line, and without overwriting work?

Update - Minimal Race Condition

The script uses the following locking mechanism:

  # Convert FLAC to MP3 using tags from flac file.
  #
  if [ ! -e $FLAC.lock ]; then
    touch $FLAC.lock
    flac -dc "$FLAC" | lame${lame_opts} \
      --tt "$TITLE" \
      --tn "$TRACKNUMBER" \
      --tg "$GENRE" \
      --ty "$DATE" \
      --ta "$ARTIST" \
      --tl "$ALBUM" \
      --add-id3v2 \
      - "$MP3"
    rm $FLAC.lock
  fi;

However, this still leaves a race condition.

Answer 1

You could implement locking of FLAC files that it's working on. Something like:

if (not flac locked)
  lock flac
  do work
else
  continue to next flac

Answer 2

To lock a file process you can create a file with the same name with a .lock extension.

Before starting the encoding check the existence of the .lock file, and optionally make sure the date of the lockfile isn't too old (in case the process dies). If it does not exist, create it before the encoding starts, and remove it after the encoding is complete.

You can also flock the file, but this only really works in c where you are calling flock() and writing to the file then closing and unlocking. For a shell script, you probably are calling another utility to do the writing of the file.

Answer 3

How about writing a Makefile?

ALL_FLAC=$(wildcard *.flac)
ALL_MP3=$(patsubst %.flac, %.mp3, $(ALL_FLAC)
all: $(ALL_MP3)
%.mp3: %.flac
        $(FLAC) ...

Then do

$ make -j4 all

Answer 4

In bash it's possible to set noclobber option to avoid file overwriting.

help set | egrep 'noclobber|-C'

Answer 5

Send output to a temporary file with a unique name, then rename the file to the desired name.

flac -dc "$FLAC" | lame${lame_opts} \
      --tt "$TITLE" \
      --tn "$TRACKNUMBER" \
      --tg "$GENRE" \
      --ty "$DATE" \
      --ta "$ARTIST" \
      --tl "$ALBUM" \
      --add-id3v2 \
      - "$MP3.$$"
mv "$MP3.$$" "$MP3"

If a race condition leaks through your file locking system every once in a while, the final output will still be the result of one process.

Answer 6

The "lockfile" command provides what you're trying to do for shell scripts without the race condition. The command was written by the procmail folks specifically for this sort of purpose and is available on most BSD/Linux systems (as procmail is available for most environments).

Your test becomes something like this:

lockfile -r 3 $FLAC.lock
if test $? -eq 0 ; then
  flac -dc "$FLAC" | lame${lame_opts} \
    --tt "$TITLE" \
    --tn "$TRACKNUMBER" \
    --tg "$GENRE" \
    --ty "$DATE" \
    --ta "$ARTIST" \
    --tl "$ALBUM" \
    --add-id3v2 \
    - "$MP3"
fi
rm -f $FLAC.lock

Alternatively, you could make lockfile keep retrying indefinitely so you don't need to test the return code, and instead can test for the output file for determining whether to run flac.