Importing HTML using creates odd characters from non-standard characters?

Question

We're attempting to read in an HTML file that contains certain MS Word characters (such as that long hyphen). The problem is these characters, for example, are showing up as garbage in SQL 2008. The data column is varbinary, and am viewing this data by casting to varchar. Here is the code, verbatim:

EDIT: Corrected definition of bad characters

var file = new FileInfo(/*file info*/);

using (var fs = file.OpenRead())
{
    var buffer = new byte[16 * 1024];
    using (var ms = new MemoryStream())
    {
        int read;
        while ((read = fs.Read(buffer, 0, buffer.Length)) > 0)
        {
            ms.Write(buffer, 0, read);
        }
        item.Data = ms.ToArray();
    }
}

The "item" object is outside the scope of the code.

If it makes any different, we are using EF 4. The data type for this data column in question is binary. Please let me know what code or details I can provide. Thanks.

Answer 1

Casting arbitrary bytes into some arbitrary code page shows up as funky characters. Nothing new here, this was always the case and will always be. You need to properly manage your text endcoding end-to-end, from the file being read to the final data being shown. Start by reading this: International Features in Microsoft SQL Server 2005. This old KB is also helpfull (in some way at least) Description of storing UTF-8 data in SQL Server. Once you figure out what encoding your HTML files are and what encoding do you whish to display the data, we can discuss available options.

Oh, and I forgot the obligatory link: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

Answer 2

As a temporary solution, if I am not wrong, the characters are shown like a square, no? You can always substitute the annoying characters once you display them.

You look for the ASCII code (to know it, you only have to convert.int32) and you replace it with the character you prefer.