How do I apply a shell command to many files in nested (and poorly escaped) subdirectories?

Question

Hi! I'm trying to do something like the following:

for file in `find . *.foo`
do
somecommand $file
done

But the command isn't working because $file is very odd. Because my directory tree has crappy file names (including spaces), I need to escape the find command. But none of the obvious escapes seem to work: -ls gives me the space-delimited filename fragments -fprint doesn't do any better.

I also tried: for file in "find . *.foo -ls"; do echo $file; done - but that gives all of the responses from find in one long line.

Any hints? I'm happy for any workaround, but am frustrated that I can't figure this out.

Thanks, Alex

(Hi Matt!)

Answer 1

Instead of relying on the shell to do that work, rely on find to do it:

find . -name "*.foo" -exec somecommand "{}" \;

Then the file name will be properly escaped, and never interpreted by the shell.

Answer 2

xargs is your friend. You will also want to investigate the -0 (zero) option with it. find (with -print0) will help to produce the list. The Wikipedia page has some good examples.

Another useful reason to use xargs, is that if you have many files (dozens or more), xargs will split them up into individual calls to whatever xargs is then called upon to run (in the first wikipedia example, rm)

Answer 3

find . -name '*.foo' -print0 | xargs -0 -n 1 somecommand

It does get messy if you need to run a number of shell commands on each item, though.

Answer 4

find . -name '*.foo' -print0 | xargs -0 sh -c 'for F in "${@}"; do ...; done' "${0}"

Answer 5

You have plenty of answers that explain well how to do it; but for the sake of completion I'll repeat and add to it:

xargs is only ever useful for interactive use (when you know all your filenames are plain - no spaces or quotes) or when used with the -0 option. Otherwise, it'll break everything.

find is a very useful tool; put using it to pipe filenames into xargs (even with -0) is rather convoluted as find can do it all itself with either -exec command {} \; or -exec command {} + depending on what you want:

find /path -name 'pattern' -exec somecommand {} \;
find /path -name 'pattern' -exec somecommand {} +

The former runs somecommand with one argument for each file recursively in /path that matches pattern.

The latter runs somecommand with as many arguments as fit on the command line at once for files recursively in /path that match pattern.

Which one to use depends on somecommand. If it can take multiple filename arguments (like rm, grep, etc.) then the latter option is faster (since you run somecommand far less often). If somecommand takes only one argument then you need the former solution. So look at somecommand's man page.

More on find: http://mywiki.wooledge.org/UsingFind

In bash, for is a statement that iterates over arguments. If you do something like this:

for foo in "$bar"

you're giving for one argument to iterate over (note the quotes!). If you do something like this:

for foo in $bar

you're asking bash to take the contents of bar and tear it apart wherever there are spaces, tabs or newlines (technically, whatever characters are in IFS) and use the pieces of that operation as arguments to for. That is NOT filenames. Assuming that the result of a tearing long string that contains filenames apart wherever there is whitespace yields in a pile of filenames is just wrong. As you have just noticed.

The answer is: Don't use for, it's obviously the wrong tool. The above find commands all assume that somecommand is an executable in PATH. If it's a bash statement, you'll need this construct instead (iterates over find's output, like you tried, but safely):

while read -r -d ''; do
    somebashstatement "$REPLY"
done < <(find /path -name 'pattern' -print0)

This uses a while-read loop that reads parts of the string find outputs until it reaches a NULL byte (which is what -print0 uses to separate the filenames). Since NULL bytes can't be part of filenames (unlike spaces, tabs and newlines) this is a safe operation.

If you don't need somebashstatement to be part of your script (eg. it doesn't change the script environment by keeping a counter or setting a variable or some such) then you can still use find's -exec to run your bash statement:

find /path -name 'pattern' -exec bash -c 'somebashstatement "$1"' -- {} \;
find /path -name 'pattern' -exec bash -c 'for file; do somebashstatement "$file"; done' -- {} +

Here, the -exec executes a bash command with three or more arguments.

1. The bash statement to execute.
2. A --. bash will put this in $0, you can put anything you like here, really.
3. Your filename or filenames (depending on whether you used {} \; or {} + respectively). The filename(s) end(s) up in $1 (and $2, $3, ... if there's more than one, of course).

The bash statement in the first find command here runs somebashstatement with the filename as argument.

The bash statement in the second find command here runs a for(!) loop that iterates over each positional parameter (that's what the reduced for syntax - for foo; do - does) and runs a somebashstatement with the filename as argument. The difference here between the very first find statement I showed with -exec {} + is that we run only one bash process for lots of filenames but still one somebashstatement for each of those filenames.

All this is also well explained in the UsingFind page linked above.