saxparser ignore byte order mark

Question

Hi,

Our saxparser does not ignore the byte order mark which appears at the starting of the file.

How do I get my sax parser to ignore the byte order mark ?

Answer 1

It looks like you may be giving utf-16 input to a saxparser which doesn't expect utf-16. Try to convert the data into utf-8, it might help.

Answer 2

Check the file in a hex editor.

If the initial bytes are indeed \xEF\xBB\xBF followed by the document itself, then it's a UTF-8 faux-BOM. Although UTF-8 faux-BOMs are an foul wrongness and tools that generate them need to be destroyed with special programming acid, the XML spec does require parsers to recognise and ignore this byte sequence, so if your SAX parser doesn't it's not compliant and needs some kicking.

If the initial bytes that are giving you  are actually something like one of:

\xC3\xAF\xC2\xBB\xC2\xBF 
\xEF\xBB\xBF\xC3\xAF\xC2\xBB\xC2\xBF 
\xEF\x00\xBB\x00\xBF\x00
\xFF\xFE\xEF\x00\xBB\x00\xBF\x00

then what you've got is an accidental double-encoding. In this case, you need to look at the program producing the file because it's not well-formed, a SAX parser would be correct to complain, and other Unicode characters in the file would probably be messed up too. Possibly it is doing something silly like serialising the document to a byte string then sending it through a bogus decode/encode cycle.

Either way, if you need to have the parser skip the troublesome byte sequence, you would have to feed it with a version you'd manually hacked to remove this prefix. Without knowing what you SAX parser is (or even what language) it's difficult to say how to do this.

Maybe you can seek the input stream before passing it to the parser? Maybe you can read the file into a byte string and pass that, shorn of the initial bytes, to the parser? If your parser doesn't give you those options you would have to load the file in as bytes, clip the beginning and save it out again to a new file.