How Do I grep For non-ASCII Characters in UNIX

Question

I have several very large XML files and I'm trying to find the lines that contain non-ASCII characters. I've tried the following:

grep -e "[\x{00FF}-\x{FFFF}]" file.xml

But this returns every line in the file, regardless of whether the line contains a character in the range specified.

Do I have the syntax wrong or am I doing something else wrong? I've also tried:

egrep "[\x{00FF}-\x{FFFF}]" file.xml 

(with both single and double quotes surrounding the pattern).

Answer 1

Strangely, I had to do this today! I ended up using Perl because I couldn't get grep/egrep to work (even in -P mode). Something like:

cat blah | perl -en '/\xCA\xFE\xBA\xBE/ && print "found"'

Answer 2

The following works for me:

grep -P "[\x80-\xFF]" file.xml

Non-ASCII characters start at 0x80 and go to 0xFF when looking at bytes. Grep (and family) don't do Unicode processing to merge multi-byte characters into a single entity for regex matching as you seem to want. The -P option in my grep allows the use of \xdd escapes in character classes to accomplish what you want.

Answer 3

The easy way is to define a non-ASCII character... as a character that is not an ASCII character.

LC_COLLATE=C grep '[^ -~]' file.xml

Add a tab after the ^ if necessary.

(The LC_COLLATE=C at the beginning is to avoid nasty surprises about the meaning of character ranges in many locales.)