Convert XML document from Latin1 to UTF8 using Java

Question

I am trying to create an XML document (rss feed) and have worked out all the kinks in it except for one character encoding issue. The problem is that I am using a UTF-8 encoding like so <?xml version="1.0" encoding="UTF-8"?> except the document itself is not encoded to UTF-8.

I am using the org.apache.ecs.xml package to create all the tags. I then use doc.output(stream) to write the content. This method does not seem to write output using UTF-8 and I don't know how to make that happen. Until I do, some symbols (the british pound is what I first noticed) aren't rendered properly in most readers.

--Updated with more information--

I ended up using a bad solution (as explained in the comments) to fix this problem. The correct answer seems to be dont use the org.apache.ecs.xml library. Thank you all for the help. StackOverflow wins again.

Answer 1

I'm not familiar with this package but from the source on the web I suspect it may be broken:

http://kickjava.com/src/org/apache/ecs/xml/XMLDocument.java.htm

contains stuff like

        for (int i=0; i<prolog.size(); i++) {
268             ConcreteElement e = (ConcreteElement)prolog.elementAt(i);
269             e.output(out);
270             // XXX really this should use line separator!
271 // XXX should also probably check for pretty print
272 // XXX also probably have difficulties with encoding

which suggests problems.

We use XOM (http://www.xom.nu) and that specifically has a setEncoding() on its Serializer so I would suggest changing packages...

Answer 2

Here is a function I wrote to convert all non-ASCII characters to their corresponding entity. Might help you sanitizing some PCDATA content before output.

/**
 * Creates xml entities for non ascii characters in the given String.
 */
public static String xmlEntitify(String in){

 StringBuffer b = new StringBuffer();

 for (int i=0;i<in.length();i++){

  Character c = in.charAt(i);
  if (c<128){
   b.append(c);
  }
  else if (c=='\ufeff'){
   // BOM character, just remove it
  }
  else {
   String cstr = Integer.toHexString(c).toUpperCase();
   while(cstr.length()<4){
    cstr="0"+cstr;
   }
   b.append("&#x");
   b.append(cstr);
   b.append(";");
  }
 }
 return b.toString();
}

Read your input stream into a String content, and write into the output stream xmlEntitify(content).

Your output is guaranteed to contain only ASCII characters, no more encoding problem.

UPDATE

Given the comments, I'll be even bolder : if you are not sanitizing your data, you are calling for trouble. I guess you are at least already replacing the < and & characters in your PCDATA. If not, you definitely should. I have another version of the above method which, instead of the first if, has :

if (c<128 && c!='&' && c!='<' && c!='>' &&  c!='"'){
 b.append(c);
}

so that these characters are also converted to their corresponding Unicode entity. This converts all of my PCDATA to unicode-friendly ASCII-only strings. I had no more encoding problem since I'm using this technique. I don't ever output XML PCDATA which has not been passed through this method : this is not sweeping the elephant under the carpet. It is just getting rid of the problem by being as generic as can be.

Answer 3

The simplest workaround is probably going to be changing your code like follows:

XMLDocument doc = new XMLDocument(1.0,false,Charset.defaultCharset().toString());

I'm guessing they're just using the default encoding to write characters to the stream. So pass the default encoding to the prologue and you should be fine.

I'll agree with other posters that this is probably the least of your worries. Looking at the source repository for ECS, it doesn't appear to have been updated for four years (the "ECS2" repository likewise).

And some self-promotion: if you're looking to build XML documents using a simple interface, the Practical XML library has a builder. It uses the standard JDK serialization mechanism for output.

Answer 4

Any chance you can write to a Writer rather than an OutputStream... that way you could specify the encoding.

Answer 5

Here is a solution my co-worker came up with that I THINK is the correct way to do it but what do I know. Instead of using doc.output(stream) we used:

    try {
            IOUtils.write(doc.toString(), stream, "UTF-8");
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

To be honest I dont completely understand the problem yet, which is why I am having problems in the first place. It seems that @subtenante's solution went through and converted any character that UTF-8 could not represent and replaced it with the unicode entity. This solution seems to write to the stream using the UTF-8 encoding like I originally wanted doc.output to. I dont know the exact difference, just that both solved my problems. Any further comments to help me understand the problem would be appreciated.