how to add encoding information to the response stream ?

Question

hi

i have following piece of code:

public void ProcessRequest (HttpContext context) 
{
    context.Response.ContentType = "text/rtf; charset=UTF-8";
    context.Response.Charset = "UTF-8";
    context.Response.ContentEncoding = System.Text.Encoding.UTF8;
    context.Response.AddHeader("Content-disposition", "attachment;filename=lista_obecnosci.csv");
    context.Response.Write("ąęćżźń󳥌ŻŹĆŃŁÓĘ");
}

When i try to open generated csv file, i get following behaviour:
- in notepad2 - everything is fine
- in word - conversion wizzard opens, and asks to convert the text. it suggest utf-8, which is somehow ok
- in excel - i get real mess. none of those polish characters can be displayed.

i wanted to write those special encoding-informatio characters in front of my string - ie.

context.Response.Write((char)0xef);
context.Response.Write((char)0xbb);
context.Response.Write((char)0xbf);

but that won't do any good. the response stream is treating that as normal data and converts it to something different.

I'd appreciate help on this one...

Greg

Answer 1

What you call "encoding-information" is actually a BOM. I suspect each of those "characters" is getting encoded separately. To write the BOM manually, you have to write it as three bytes, not three characters. I'm not familiar with the .NET I/O classes, but there should be a method available to you that takes a byte or byte[] parameter and writes them directly to the file.

By the way, the UTF-8 BOM is optional; in fact, its use is discouraged by the Unicode Consortium. If you don't have a specific reason for using it, save yourself some hassle leave it out.

EDIT: I just remembered you can also write the actual BOM character, '\uFEFF', and let the encoder handle it:

context.Response.Write('\uFEFF');

Answer 2

I think the problem is with Excel based on Microsoft Excel mangles Diacritics in .csv files. To prove this, copy your sample output string of ąęćżźń󳥌ŻŹĆŃŁÓĘ and paste into a test file using your favorite editor, and save as a UTF-8 encoded .csv file. Open in Excel and see the same issues.