Elegant way to search for UTF-8 files with BOM?

Answer 1

If you accept some false positives (in case there are non-text files, or in the unlikely case there is a ZWNBSP in the middle of a file), you can use grep:

fgrep -rl `echo -ne '\xef\xbb\xbf'` .

Answer 2

find -type f -print0 | xargs -0 grep -l `printf '^\xef\xbb\xbf'` | sed 's/^/found BOM in: /'

• find -print0 puts a null \0 between each file name instead of using new lines
• xargs -0 expects null separated arguments instead of line separated
• grep -l lists the files which match the regex
• The regex ^\xeff\xbb\xbf isn't entirely correct, as it will match non-BOMed UTF-8 files if they have zero width spaces at the start of a line

Answer 3

I would use something like:

grep -orHbm1 "^`echo -ne '\xef\xbb\xbf'`" . | sed '/:0:/!d;s/:0:.*//'

Which will ensure that the BOM occurs starting at the first byte of the file.

Answer 4

What about this one simple command which not just finds but clears nasty BOM? :)

find . -type f -exec sed 's/^\xEF\xBB\xBF//' -i.bak {} \; -exec rm {}.bak \;

I love "find" :)

If you want just to show BOM files, use this one:

grep -rl $'\xEF\xBB\xBF' .

Answer 5

find . -type f -print0 | xargs -0r awk '
    /^\xEF\xBB\xBF/ {print FILENAME}
    {nextfile}'

Most of the solutions given above test more than the first line of the file, even if some (such as Marcus's solution) then filter the results. This solution only tests the first line of each file so it should be a bit quicker.