Unix Find Replace Special Characters in Multiple Files

Question

I've got a set of files in a web root that all contain special characters that I'd like to remove (Â,€,â,etc).

My command

find . -type f -name '*.*' -exec grep -il "Â" {} \;

finds & lists out the files just fine, but my command

find . -type f -name '*.*' -exec tr -d 'Â' '' \;

doesn't produce the results I'm looking for.

Any thoughts?

Answer 1

Use

tr -d 'Â'

What does the ' ' stands for? On my system using your command produces this error:

tr: extra operand `'

Only one string may be given when deleting without squeezing repeats.

Try `tr --help' for more information.

Answer 2

I would recommend looking into sed. It can be used to replace the contents of the file.

So you could use the command:

find . -type f -name '*.*' -exec sed -i "s/Â//" {} \;

I have tested this with a simple example and it seems to work. The -exec should handle files with whitespace in their name, but there may be other vulnerabilities I'm not aware of.

Answer 3

sed 's/ø//' file.txt

That should do the trick for replacing a special char with an empty string.

find . -name "*.*" -exec sed 's/ø//' {} \

Answer 4

to replace all non-ascii characters in all files inside the current directory you could use:

find . -type f | xargs perl -pi.bak -e 's,[^[:ascii:]],,g'

afterwards you will have to find and remove all the '.bak' files:

find . -type f -a -name \*.bak | xargs rm

Answer 5

I would use something like this.

for file in `find . -type f`
do
    # Search for char end remove it. Save file as file.new
    sed -e 's/[ۉ]//g' $file > $file.new
    # mv file.new to file DON'T RUN IF YOU WILL NOT OVERITE ORIGINAL FILE
    mv $file.new $file
done

The above script will fail as levislevis85 has mentioned it with spaces in filenames. This would not be the case if you use the following code.

find . -type f | while read file
do 
    # Search for char end remove it. Save file as file.new
    sed -e 's/[ۉ]//g' "$file" > "$file".new
    # mv file.new to file DON'T RUN IF YOU WILL NOT OVERITE ORIGINAL FILE
    mv "$file".new "$file"
done

Answer 6

It would be helpful to know what "doesn't produce the results I'm looking for" means. However, in your command tr is not provided with the filenames to process. You could change it to this:

find . -type f -name '*.*' -exec tr -d 'Â' {} \;

Which is going to output everything to stdout. You probably want to modify the files instead. You can use Grundlefleck's answer, but one of the issues alluded to in that answer is if there are large numbers of files. You can do this:

find . -type f -name '*.*' -print0 | xargs -0 -I{} sed -i "s/Â//" \{\}

which should handle files with spaces in their names as well as large numbers of files.

Answer 7

with bash shell

for file in *.*
do
  case "$file" in 
   *[^[:ascii:]]* )
         mv "$file" "${file//[^[:ascii:]]/}"
   ;;
  esac
done