ubuntu/linux bash: traverse directory and subdirectories to work with files

Question

let me start off with what I need; the program is given a directory, it will then examine all the files in the directory (works) and do stuff to the files (waiting till it can find all the files for this part). then it will look for subdirectories and re-run its self for each subdirectory.

the directory I'm testing with looks like this:

desktop/test_files/ (starting directory)
desktop/test_files/folder 1/
desktop/test_files/folder 1>folder 2/
desktop/test_files/folder 1>folder 2/<files, 20 or so>
desktop/test_files/folder 3/
desktop/test_files/folder 3/<more files, 20 or so>

folders and files do contain spaces in the names

the output is:

$ ./x007_shorter.sh Desktop/test_files/

Desktop/test_files/"folder 1"/
Desktop/test_files/folder 1/"folder 2"/
ls: cannot access */: No such file or directory
Desktop/test_files/folder 1/folder 2/"folder 3"/
./x007_shorter.sh: line 4: cd: ./folder 3/: No such file or directory
ls: cannot access */: No such file or directory

here is the program:

#!/bin/bash
function findir {
    newDir=$1
    eval cd $newDir
    ARRAY=( $(ls -d */) )
    declare -a diry
    count=0
    a=0
    while [ $a -lt ${#ARRAY[@]} ]; do
        diry[$count]="${ARRAY[$a]}"
        noSpace=true
        while [ true ]; do
            if [[ ${diry[$count]} == */* ]] ; then
                if [ $noSpace = false ]; then
                diry[$count]="${diry[$count]:0:((${#diry[$count]}-1))}\"/"
                fi
                break
                noSpace=true
            fi
            let "a=$a+1"
            if [ $noSpace = false ]; then
                diry[$count]="${diry[$count]} ${ARRAY[$a]}"
            else
                diry[$count]="\"${diry[$count]} ${ARRAY[$a]}"
            fi
            noSpace=false
        done
        let "count=$count+1"
        let "a=$a+1"
    done
    for a in `seq 1 ${#diry[@]}`; do
        eval cd .$newDir
#        list "${diry[($a-1)]}"
        where=`pwd`
#        eval cd $newDir
        #findir "${diry[($a-1)]}"
        #findir "$where${diry[($a-1)]:1}"
        #Right option won,  echo "${diry[($a-1)]} Vs $where/${diry[($a-1)]}"
        echo "$where/${diry[($a-1)]}"
        findir "./${diry[($a-1)]}"
    done
}
function list {
    input_file_directory=$1
    eval cd $input_file_directory
    ARRAY=( $(find . -maxdepth 1 -type f -print) )
    declare -a files
    count=0
    a=0
    while [ $a -lt ${#ARRAY[@]} ]; do
        files[$count]="${ARRAY[$a]}"
        while [ true ]; do
            if [[ ${ARRAY[(($a+1))]} == ./* ]] ; then
                break
            fi
            if [[ "${ARRAY[(($a+1))]}" == "" ]] ; then
                break
            fi
            let "a=$a+1"
            files[$count]="${files[$count]} ${ARRAY[$a]}"
        done
        let "count=$count+1"
        let "a=$a+1"
    done
    where=`pwd`
    for a in `seq 1 ${#files[@]}`; do
        echo "$where${files[($a-1)]:1}"
        #going to work on each file, just echoing file till lists all files
    done
}

clear
dar=""
if [[ $1 = "" ]]; then
    read -p "Please enter a directory for me to scan" newdir
    dar=$newdir
    list $newdir
    findir $newdir
else
    dar=$1
    list $1
    findir $1
fi

Answer 1

Any reason you can't use find for this? Stick the per-file operation you want in it's own script (I've called it dostufftomyfile.sh below), then do:

find $dir -type f -print0 | xargs -0 dostufftomyfile.sh

Replacing $dir with the top level directory you'll be searching from...

Edited to add... When you write the shell script, make sure you put $@ in double-quotes... e.g., you'll want your dostufftomyfile.sh script to have this structure:

#!/bin/sh
for f in "$@"
do
    echo "Processing file: $f"
    # Do something to file $f
done

if you don't quote $@ then the spaces in filenames will be ignored (which I suspect you won't want) :-)

Answer 2

Chris J's answer is the preferred way to do things if you can put the per-file stuff in a separate command(/script). If you want everything in a single script, my favorite incantation is something like this:

while IFS="" read -r -d $'\000' file <&3; do
    dostuffwith "$file"
done 3< <(find -x  "$dir" -mindepth 1 -type f -print0)

See BashFAQ #20 and #89 for explanations and some other options. Note that this only works in bash (i.e. the script must start with #!/bin/bash). Also, it processes the contents of a given directory in alphabetic order, rather than files-before-subdirectories.

If you really want to step through the files "by hand" (i.e. to get more control over the traversal order), here's how I'd do it:

#!/bin/bash

process_dir() {
    local -a subdirs=()
    echo "Scanning directory: $1"

    # Scan the directory, processing files and collecting subdirs
    for file in "$1"/*; do
        if [[ -f "$file" ]]; then
            echo "Processing file: $file"
            # actually deal with the file here...
        elif [[ -d "$file" ]]; then
            subdirs+=("$file")
            # If you don't care about processing all files before subfolders, just do:
            # process_dir "$file"
        fi
    done

    # Now go through the subdirs
    for d in "${subdirs[@]}"; do
        process_dir "$d"
    done
}

clear
if [[ -z "$1" ]]; then
    read -p "Please enter a directory for me to scan " dir
else
    dir="$1"
fi
process_dir "$dir"

Answer 3

You have the error "No such file .... due to this

ARRAY=( $(ls -d */) )

When its expanded, directories with whitespaces will get stored in array as individual elements. eg Desktop/test_files/folder 1/folder 2/"folder 3"/.

In the array, element 0 will be Desktop/test_files/folder, element 1 will be 1/folder and so on. That's why your script can't find the directory.

You can set the IFS to $'\n' before assigning to the array

OLDIFS=$IFS
IFS=$'\n'
ARRAY=($(ls -d */))
IFS="$OLDIFS"