Stripping Invalid XML characters in Java

Question

I have an XML file that's the output from a database. I'm using the Java SAX parser to parse the XML and output it in a different format. The XML contains some invalid characters and the parser is throwing errors like 'Invalid Unicode character (0x5)'

Is there a good way to strip all these characters out besides pre-processing the file line-by-line and replacing them? So far I've run into 3 different invalid characters (0x5, 0x6 and 0x7). It's a ~4gb database dump and we're going to be processing it a bunch of times, so having to wait an extra 30 minutes each time we get a new dump to run a pre-processor on it is going to be a pain, and this isn't the first time I've run into this issue.

Answer 1

If the file contains invalid characters, it isn't an XML file. Ask the creators of it to create only well-formed XML in future.

I've had this problem a lot in the past. People don't seem to understand that XML needs to be well-formed and not contain rubbish.

Answer 2

I haven't used this personally but Atlassian made a command line XML cleaner that may suit your needs (it was made mainly for JIRA but XML is XML):

Download atlassian-xml-cleaner-0.1.jar

Open a DOS console or shell, and locate the XML or ZIP backup file on your computer, here assumed to be called data.xml

Run: java -jar atlassian-xml-cleaner-0.1.jar data.xml > data-clean.xml

This will write a copy of data.xml to data-clean.xml, with invalid characters removed.

Answer 3

Is it possible your invalid characters are present only within the values and not the tags themselves i.e. the XML notionally meets the schema but the values have not been properly sanitized? If so, what about overriding InputStream to create a CleansingInputStream that replaces your invalid characters with their XML equivalents?

Answer 4

Your problem does not concern XML: it concerns character encodings. What it comes down to is that every string, be it XML or otherwise, consists of bytes and you cannot know what characters these bytes represent, unless you are told what character encoding the string has. If, for instance, the supplier tells you it's UTF-8 and it's actually something else, you are bound to run into problems. In the best case, everything works, but some bytes are translated into 'wrong' characters. In the worst case you get errors like the one you encountered.

Actually, your problem is even worse: your string contains byte sequences that do not represent characters in any character encoding. There is no texthandling tool, let alone an XML parser, that can help you here. This needs byte-level cleaning up.

Answer 5

There is a Java code snippet here to strip invalid XML characters:

http://cse-mjmcl.cse.bris.ac.uk/blog/2007/02/14/1171465494443.html