C# UTF-8 Encoding Problem

Question

I've searched posts here on Stack Overflow, and read JoelOnSoftware's post on encoding, and now have a basic grasp of encoding issues. But I'm running into a problem with some character encoding coming from the Windows clipboard.

The reproducible test is to use IE and select and copy the "Advertising Programs" text from the Google homepage.

I'm using the following C# code to pull this text off the clipboard (error checking removed):

uint FormatId = GetRegisteredClipboardFormatId("HTML Format");
IntPtr hHtml = Win32.GetClipboardData(FormatId);
uint DataSize = Win32.GlobalSize(hHtml);
byte[] HtmlData = new byte[DataSize];
IntPtr pData = Win32.GlobalLock(hHtml);
Marshal.Copy(pData, HtmlData, 0, (int)DataSize);
Win32.GlobalUnlock(hHtml);

The clipboard HTML data is supposed to be UTF-8 encoded, so I use the following to convert the data to a string:

string Content = Encoding.UTF8.GetString(HtmlData);

However, ignoring the surrounding HTML tags, what this results in is:

"Advertising Programs"

Am I doing something wrong, misunderstanding something, or does the problem lie elsewhere?

Thanks for any help!

Answer 1

Check the HTML code. There is " " between "Advertising" and "Programs".

Try your code with "Business Solutions" text and it will work.

Most probably you will need to replace nbsp with a normal space.

Answer 2

You are displaying UTF-8 as Latin-1 or its variants (CP1252).

Google uses a nbsp in that sentence, which is C2 A0, which happens to be " " in Latin-1.

EDIT: The code you showed here is ok. I think the problem occurs when you display the content. Looks like you are output UTF-8 but the display media is expecting Latin-1.

If you are using console to display, try this,

 Console.OutputEncoding = Encoding.GetEncoding("iso-8859-1");

This will tell console to send out Latin-1, instead of UTF-8.

If you display the text in browsers, make sure the web page is marked with UTF-8, like,

   <meta http-equiv="Content-Type" content="text/html; charset=utf-8">