Why is my ATOM XML UTF-16 and not UTF-8

Question

Hello,

I have this code to create an ATOM feed

Dim xmlResult As New StringBuilder
Dim settings As New XmlWriterSettings
Dim atomWriter As XmlWriter = XmlWriter.Create(xmlResult, settings)
Dim atomFormatter As Atom10FeedFormatter = New Atom10FeedFormatter(feed)
atomFormatter.WriteTo(atomWriter)
atomWriter.Close()

It returns XML that starts like this:

<?xml version="1.0" encoding="utf-16"?><feed xmlns="http://www.w3.org/2005/Atom"&gt;

No matter what I try I can't get it to return utf-8 instead of utf-16. When it is utf-16 it fails to work with IE8 but utf-8 works OK. How do I know it works OK I hear you ask if I can't get it to return utf-8, well I end up using this line of code to return the feed to the browser:

Response.Write(Replace(xmlResult.ToString, "utf-16", "utf-8"))

Which is surely the worst hack you've seen today, if it isn't I feel sorry for you!

So this is what I have tried to get UTF-8:

Response.ContentEncoding = System.Text.Encoding.UTF8
Response.Charset = "UTF-8"
settings.Encoding = System.Text.Encoding.UTF8

The settings line is what I thought would work but isn't for me. My understanding is that the XML Writer is the part that is adding the XML declaration but as the settings.Encoding doesn't work I'm stumped. Please help!

Thanks

Answer 1

Try "settings.Encoding = System.Text.Encoding.UTF8"

I have tried it and, as MrBrutal noticed, it does not work.

My guess it that because the string in .NET is always UTF-16, writing an XML into a string will always force the encoding to utf-16, no matter what.

You should try writing to a stream, where you have more control. If the goal is to write into a file, then the stream should be connected to that and write directly there.

If writing to a file is not the ultimate goal, then what is is? Might help a bit if you explain a bit the use case. Maybe the final result belongs in a byte array, not a string.

Answer 2

The underlying reason is that when you call XmlWriter.Create with a StringBuilder, it creates a new StringWriter instance, which always has an encoding of UTF-16, and creates an XmlWriter around that StringWriter. Looking in Reflector, it looks like the encoding of StringWriter is a private, static variable, so all StringWriter instances will have the same encoding.

Is there some reason you're not creating an XmlWriter that directly writes to the Response OutputStream?

If you can do that, this code works for me:

With Response
  .ContentEncoding = Encoding.UTF8
  .ContentType = "text/xml"
End With

Dim atomFormatter As Atom10FeedFormatter = New Atom10FeedFormatter(feed)
Dim settings As XmlWriterSettings = New XmlWriterSettings

With settings
  .OmitXmlDeclaration = False
  .Encoding = Encoding.UTF8
End With

Using atomWriter As XmlWriter = XmlWriter.Create(Response.OutputStream, settings)
  atomFormatter.WriteTo(atomWriter)
  atomWriter.Close()
End Using

Response.End()

If you must use StringBuilder for a good reason, perhaps you might consider omitting the Xml Declaration and adding it in your code before the atomFormatter.WriteTo call. Alternatively, you could write your function to accept an arbitrary Stream and create an XmlWriter on that, using a MemoryStream or FileStream when you aren't planning to write directly to the Response.OutputStream.

Apologies if the code is not idiomatic VB; I mostly write in C#.