Randomizing arg order for a bash for statement

Question

I have a bash script that processes all of the files in a directory using a loop like

for i in *.txt do ops..... done

There are thousands of files and they are always processed in alphanumerical order because of '*.txt' expansion.

Is there a simple way to random the order and still insure that I process all of the files only once?

Answer 1

You could pipe your filenames through the sort command:

ls | sort --random-sort | xargs ....

Answer 2

Here's an answer that relies on very basic functions within awk so it should be portable between unices.

ls -1 | awk '{print rand()*100, $0}' | sort -n | awk '{print $2}'

EDIT:

ephemient makes a good point that the above is not space-safe. Here's a version that is:

ls -1 | awk '{print rand()*100, $0}' | sort -n | sed 's/[0-9\.]* //'

Answer 3

Assuming the filenames do not have spaces, just substitute the output of List::Util::shuffle.

for i in `perl -MList::Util=shuffle -e'$,=$";print shuffle<*.txt>'`; do
    ....
done

If filenames do have spaces but don't have embedded newlines or backslashes, read a line at a time.

perl -MList::Util=shuffle -le'$,=$\;print shuffle<*.txt>' | while read i; do
    ....
done

To be completely safe in Bash, use NUL-terminated strings.

perl -MList::Util=shuffle -0 -le'$,=$\;print shuffle<*.txt>' |
while read -r -d '' i; do
    ....
done

---

Not very efficient, but it is possible to do this in pure Bash if desired. sort -R does something like this, internally.

declare -a a                     # create an integer-indexed associative array
for i in *.txt; do
    j=$RANDOM                    # find an unused slot
    while [[ -n ${a[$j]} ]]; do
        j=$RANDOM
    done
    a[$j]=$i                     # fill that slot
done
for i in "${a[@]}"; do           # iterate in index order (which is random)
    ....
done

Or use a traditional Fisher-Yates shuffle.

a=(*.txt)
for ((i=${#a[*]}; i>1; i--)); do
    j=$[RANDOM%i]
    tmp=${a[$j]}
    a[$j]=${a[$[i-1]]}
    a[$[i-1]]=$tmp
done
for i in "${a[@]}"; do
    ....
done

Answer 4

Here's a solution with standard unix commands:

for i in $(ls); do echo $RANDOM-$i; done | sort | cut -d- -f 2-

Answer 5

Here's a Python solution, if its available on your system

import glob
import random
files = glob.glob("*.txt")
if files:
    for file in random.shuffle(files):
        print file

Answer 6

If you have GNU coreutils, you can use shuf:

while read -d '' f
do
    # some stuff with $f
done < <(shuf -ze *)

This will work with files with spaces or newlines in their names.

Off-topic Edit:

To illustrate SiegeX's point in the comment:

$ a=42; echo "Don't Panic" | while read line; do echo $line; echo $a; a=0; echo $a; done; echo $a
Don't Panic
42
0
42
$ a=42; while read line; do echo $line; echo $a; a=0; echo $a; done < <(echo "Don't Panic"); echo $a
Don't Panic
42
0
0

The pipe causes the while to be executed in a subshell and so changes to variables in the child don't flow back to the parent.