"É" not getting converted to two bytes correctly.

Question

Further to this question I've got a supplementary problem.

I've found a track with an "É" in the title.

My code:

var playList = new StreamWriter(playlist, false, Encoding.UTF8);

-

private static void WriteUTF8(StreamWriter playList, string output)
{
    byte[] byteArray = Encoding.UTF8.GetBytes(output);
    foreach (byte b in byteArray)
    {
        playList.Write(Convert.ToChar(b));
    }
}

converts this to the following bytes:

195
137

which is being output as à followed by a square (which is an character that can't be printed in the current font).

I've exported the same file to a playlist in Media Monkey at it writes the "É" as "É" - which I'm assuming is correct (as KennyTM pointed out).

My question is, how do I get the "‰" symbol output? Do I need to select a different font and if so which one?

UPDATE

People seem to be missing the point.

I can get the "É" written to the file using

playList.WriteLine("É");

that's not the problem.

The problem is that Media Monkey requires the file to be in the following format:

#EXTINFUTF8:140,Yann Tiersen - Comptine D'Un Autre Été: L'Après Midi
#EXTINF:140,Yann Tiersen - Comptine D'Un Autre Été: L'Après Midi
#UTF8:04-Comptine D'Un Autre Été- L'Après Midi.mp3
04-Comptine D'Un Autre Été- L'Après Midi.mp3

Where all the "high-ascii" (for want of a better term) are written out as a pair of characters.

UPDATE 2

I should be getting c9 replaced by c3 89.

I was going to put what I'm actually getting, but in doing the tests for this I've managed to get a test program to output the text in the right format "as is". So I need to do some more investigation.

Answer 1

I don't do C# but the symptoms tell me that you're indeed writing it as UTF-8, but that the output/console/application/whatever with which you're viewing the written output is not using UTF-8, but ISO-8859-1 to display them and that MediaMonkey is using CP1252 to display them.

If you're viewing them in the IDE console, then you need to configure the IDE to use UTF-8 as console and text file encoding.

Update you apparently want to write UTF-8 data as CP-1252. Now the question/problem is more clear. Again, I don't do C#, but the Java equivalent would be:

Writer writer = new OutputStreamWriter(new FileOutputStream("file.ext"), "CP-1252");
writer.write(someUTF8String); // Will be written as CP-1252. "É" would become "É"

Hopefully this gives some insights.

Answer 2

Using Convert.ToChar like that is almost certainly a bad idea. You're basically encoding things twice.

You should either be performing the conversion yourself and then writing directly to a stream, or you should be letting the StreamWriter do the conversion. Why are you using a StreamWriter at all if you're trying to perform the conversions yourself?

Are you trying to write to a binary file, or a simple text file? If it's a simple text file, just use a StreamWriter and let that do the conversion. If it's a binary file, use a Stream instead of a StreamWriter, and perform text encoding directly where you need to, writing the bytes straight to the stream afterwards.

EDIT: Here's what's happening with your original code:

Encoding.UTF8.GetBytes(text) => byte[] { 0xc3, 0x89 };

Convert.ToChar(0xc3) => char U+00C3
StreamWriter writes U+00C3 as byte[] { 0xc3, 0x83 };

Convert.ToChar(0x89) => char U+0089
StreamWriter writes U+00C3 as byte[] { 0xc2, 0x89 };

So that's why you're getting c3 83 c2 89 written to the file.

Answer 3

StreamWriter already converts the characters you send it to UTF-8 — that's its entire purpose. Throw WriteUTF8 away; it's broken and useless.

(WriteUTF8 is taking characters, converting them to UTF-8 bytes, converting each single byte to the character it maps to in the current code page, then encoding each of those characters in UTF-8. So in the best case you have a doubly-UTF-8-encoded string; in the worst, you've completely lost bytes that weren't mapped in the system code page repertoire; especially bad for DBCS code pages.)

The problem you're having with Media Monkey may be just that it doesn't support UTF-8 or Unicode filenames at all. Try asking it to play (and export a playlist for) files with characters that don't fit in your system codepage, for example by renaming a file to αβγ.mp3.

Edit:

#EXTINFUTF8:140,Yann Tiersen - Comptine D'Un Autre Été: L'Après Midi
#EXTINF:140,Yann Tiersen - Comptine D'Un Autre Été: L'Après Midi
#UTF8:04-Comptine D'Un Autre Été- L'Après Midi.mp3
04-Comptine D'Un Autre Été- L'Après Midi.mp3

OK, what you've got there is a mixture of encodings in the same file: it's no wonder text editors are going to have trouble opening it. The uncommented and #EXTINF lines are in the system default code page, and are present to support media players that can't read Unicode filenames. Any filename characters not present in the system code page (eg. Greek as above, on a Western Windows install) will be mangled and unplayable for anything that doesn't know about the #UTF8 (and #EXTINFUTF8 for the description) lines.

So if this is your target format, you'll need to grab two encodings and use each in turn, something like:

private static void writePlaylistEntry(Stream playlist, string filename, int length) {
    Encoding utf8= new UTF8Encoding(false);
    Encoding ansi= Encoding.Default;
    playlist.Write(utf8.GetBytes("#EXTINFUTF8:"+length+","+filename+"\n"));
    playlist.Write(ansi.GetBytes("#EXTINF:"+length+","+filename+"\n"));
    playlist.Write(utf8.GetBytes("#UTF8:"+filename+"\n"));
    playlist.Write(ansi.GetBytes(filename+"\n"));
}

Answer 4

The more fundamental problem is in the name of the method:

 private static void WriteUTF8(...)

.M3U files aren't UTF-8. They're Latin-1 (or Windows-1252).

Instead of Encoding.UTF8, you should be using Encoding.GetEncoding(1252). Then you can just write directly to the stream, you won't need any of this conversion weirdness.

Update:

I just tried the following C# code and the resulting .M3U opens just fine in both Winamp and WMP:

static void Main(string[] args)
{
    string fileName = @"C:\Temp\Test.m3u";
    using (StreamWriter writer = new StreamWriter(fileName, false,
        Encoding.GetEncoding(1252)))
    {
        writer.WriteLine("#EXTM3U");
        writer.WriteLine("#EXTINF:140,Yann Tiersen " +
            "- Comptine D'Un Autre Été: L'Après Midi");
        writer.WriteLine("04-Comptine D'Un Autre Été- L'Après Midi.mp3");
    }
}

So, as I said - just use the right encoding to begin with. You don't need all those extra #EXTINFUTF8 and #UTF8 lines, unless it's some bizarre requirement for Media Monkey (it's definitely not part of the basic M3U spec).

Answer 5

Right, first off thanks to everyone for their help and patience.

I've finally got it working correctly. I've implemented a version of bobince's solution which is why he gets the acceptance (up-votes to everyone else). Here's my code:

var playList = new StreamWriter(playlist, false, Encoding.Default);
playList.WriteLine("#EXTM3U");

foreach (string track in tracks)
{
    // Read ID3 tags from file
    var info = new FileProperties(track);

    // Write extended info (#EXTINF:<time>,<artist> - <title>
    if (Encoding.UTF8.GetBytes(info.Artist).Length != info.Artist.Length ||
        Encoding.UTF8.GetBytes(info.Title).Length != info.Title.Length)
    {
        playList.Close();
        playList = new StreamWriter(playlist, true, Encoding.UTF8);

        playList.WriteLine(string.Format("#EXTINFUTF8:{0},{1} - {2}",
                           info.Duration, info.Artist, info.Title));

        playList.Close();
        playList = new StreamWriter(playlist, true, Encoding.Default);
    }

    playList.WriteLine(string.Format("#EXTINF:{0},{1} - {2}",
                       info.Duration, info.Artist, info.Title));

    // Write the name of the file (removing the drive letter)
    string file = Path.GetFileName(track);
    if (Encoding.UTF8.GetBytes(file).Length != file.Length)
    {
        playList.Close();
        playList = new StreamWriter(playlist, true, Encoding.UTF8);

        playList.WriteLine(string.Format("#UTF8:{0}", file));

        playList.Close();
        playList = new StreamWriter(playlist, true, Encoding.Default);
    }

    playList.WriteLine(file);
}

playList.Close();

As you can see I assume I'm not going to have to write UTF8, but when I do I close the stream and reopen it with UTF8 encoding. I then, after writing the offending line, close and reopen it with the default encoding.

Now I don't know why my previous code gave inconsistent results. Given what everyone (particularly Jon) said it should have failed all the time, or possibly worked all of the time.