views:

15666

answers:

7

Here's the problem:

In C# I'm getting information from a legacy ACCESS database. .NET converts the content of the database (in the case of this problem a string) to Unicode before handing the content to me.

How do I convert this Unicode string back to it's ASCII equivalent?


Edit
Unicode char 710 is indeed MODIFIER LETTER CIRCUMFLEX ACCENT. Here's the problem a bit more precise:

 -> (Extended) ASCII character ê (Extended ASCII 136) was inserted in the database.
 -> Either Access or the reading component in .NET converted this to U+02C6 U+0065
    (MODIFIER LETTER CIRCUMFLEX ACCENT + LATIN SMALL LETTER E)
 -> I need the (Extended) ASCII character 136 back.


Here's what I've tried (I see now why this did not work...):

string myInput = Convert.ToString(Convert.ToChar(710));
byte[] asBytes = Encoding.ASCII.GetBytes(myInput);

But this does not result in 94 but a byte with value 63...
Here's a new try but it still does not work:

byte[] bytes = Encoding.ASCII.GetBytes("ê");


Soltution
Thanks to both csgero and bzlm for pointing in the right direction I solved the problem here.

A: 

Hmm … I'm not sure which character you mean. The caret (“^”, CIRCUMFLEX ACCENT) has the same code in ASCII and Unicode (U+005E).

/EDIT: Damn, my fault. 710 (U+02C6) is actually the MODIFIER LETTER CIRCUMFLEX ACCENT. Unfortunately, this character isn't part of ASCII at all. It might look like the normal caret but it's a different character. Simple conversion won't help here. I'm not sure if .NET supports mapping of similar characters when converting from Unicode. Worth investigating, though.

Konrad Rudolph
I just edited the post to reflect what the OP meant. :-)
Chris Jester-Young
Unicode != UTF-8
OJ
OJ: What has this got to do with UTF-8?
Chris Jester-Young
@OJ, I'm aware of that. However, the code point of a character is the same in all Unicode encodings.
Konrad Rudolph
@Chris: In Konrad's original post he talked about UTF8, not Unicode.
OJ
@OJ: Ah, gotcha.
Chris Jester-Young
You're right, it is indeed MODIFIER LETTER CIRCUMFLEX ACCENT, see my edits.
Huppie
A: 

Try doing a search and replace:

myInput.Replace('ˆ', '^');
Chris Jester-Young
So what about other Unicode characters in that case? ;-)
Huppie
A: 

The value 63 is the question mark, AKA "I am not able to display this character in ASCII".

Timbo
So, you're pinpointing my problem. The questions is how DO I do this, I know the method I tried does not work.
Huppie
+2  A: 

You cannot use the default ASCII encoding (Encoding.ASCII) here, but must create the encoding with the appropriate code page using Encoding.GetEncoding(...). You might try to use code page 1252, which is a superset of ISO 8859-1.

csgero
Like so: byte[] bytes = Encoding.GetEncoding(437).GetBytes("ê");
Huppie
+1  A: 

ASCII does not define ê; the number 136 comes from the number for the circumflex in 8-bit encodings such as Windows-1252.

Can you verify that a small e with a circumflex (ê) is actually what is supposed to be stored in the Access database in this case? Perhaps U+02C6 U+0065 is the result of a conversion error, where the input is actually an e followed by a circumflex, or something else entirely. Perhaps your Access database has corrupt data in the sense that the designated encoding does not match the contents, in which case the .NET client might incorrectly parse the data (using the wrong decoder).

If this error is indeed introduced during the reading from the database, perhaps pasting some code or configuration settings might help.

In Code page 437, character number 136 is an e with a circumflex.

bzlm
Thanks! Your tip helped a lot, it was in fact codepage 437 (MS-DOS). Using Encoding.GetEncoding(437) it worked.
Huppie
+6  A: 

Okay, let's elaborate. Both csgero and bzlm pointed in the right direction.

Because of blzm's reply I looked up the Windows-1252 page on wiki and found that it's called a codepage. The wikipedia article for Code page which stated the following:

No formal standard existed for these ‘extended character sets’; IBM merely referred to the variants as code pages, as it had always done for variants of EBCDIC encodings.

This led me to codepage 437:

n ASCII-compatible code pages, the lower 128 characters maintained their standard US-ASCII values, and different pages (or sets of characters) could be made available in the upper 128 characters. DOS computers built for the North American market, for example, used code page 437, which included accented characters needed for French, German, and a few other European languages, as well as some graphical line-drawing characters.

So, codepage 437 was the codepage I was calling 'extended ASCII', it had the ê as character 136 so I looked up some other chars as well and they seem right.

csgero came with the Encoding.GetEncoding() hint, I used it to create the following statement which solves my problem:

byte[] bytes = Encoding.GetEncoding(437).GetBytes("ê");
Huppie
A: 

Using WordPress functions, read:

[solution][howto] Convert special characters to normal chars (é to e) url: [solution][howto] Convert special characters to normal chars (é to e)

which converts txt like: testáén to testaen :)

Ramon Fincken
This... must be the most worthless reply in a topic ever! I'm speechless. This already borders on heroic ingenioustness! You managed to violate at least 4 unrelated reasons why NOT to post this: The question was finished more than a year ago; the OP is working in .NET; the question is in no way whatsoever connected to Wordpress; the OP doesn't want to get rid of "bad" characters but to retain them correctly. WTF?!
Vilx-