Identifying and removing null characters in UNIX

Question

I have a text file containing unwanted null characters. When I try to view it in I see ^@ symbols, interleaved in normal text. How can I:

a) Identify which lines in the file contains null characters? I have tried grepping for \0 and \x0, but this did not work.

b) Remove the null characters? Running strings on the file cleaned it up, but I'm just wondering if this is the best way?

Thanks

Answer 1

I'd use "tr":

tr < file-with-nulls -d '\000' > file-without-nulls

except ... for reasons that escape me right now that is not working when I test it - I need to check to see if I'm in an alternate universe but I'll be back

edit nope I'm still here on Earth and dumb as ever - fixed the command so that the -d argument comes before the '\000' argument

Answer 2

A large number of unwanted NUL characters, say one every other byte, indicates that the file is encoded in UTF-16 and that you should use iconv to convert it to UTF-8.

Answer 3

Use the following sed command for removing the null characters in a file.

sed -i 's/\x0//g' null.txt

Answer 4

I discovered the following, which prints out which lines, if any, have null characters:

perl -ne '/\000/ and print;' file-with-nulls

Also, an octal dump can tell you if there are nulls:

od file-with-nulls | grep ' 000'