very strange string unique issue in C#

Answer 1

When 1 of the strings is converted to be stored in the datatable it is resulting in the same string as another. so the unique constraint on the dataTable makes it throw an exception.

Answer 2

Not sure what font or character set those cool characters are in but they don't seem to sort very well. Compare() works based on the sortability of the strings which is why culture is very important for comparing culture sensitive strings. These character strings don't return as different from a sorting perspective so they are in-effect "the same". The String.Equals() method will show them as different.

buf[0].Equals(buf[1]) = false

Not sure why the need to use the special characters but if that is a requirement to have as "unique" keys that may be an issue. I am assuming that the datatable is using a similar comparison to validate unique column values and is therefore seeing the two rows as duplicate.

Answer 3

Hi.

I saw you used InvariantCulture in your comparison. You should use Ordinal (character-by-character literal comparison) or CurrentCulture (which takes substitutions into place - like Æ === AE) to do your comparisons.

You might get better luck by entering the characters as Unicode character strings like:

string text = "\uEFBF\uBDEF\uBFBD\uEFBF";
string text2= "\uEFBF\uBD0D\uAEFB\uFBDE\uFBFB";

I've got this as some Chinese/Japanese characters (not pasting completely):

string text = "뷯뾽";
string text2 = "봍껻ﯞﯻ";

CurrentCulture will know that a single symbol can represent 2 other symbols, so it would be a good choice to use. Ordinal will just notice that the lengths are different. If they are the same length and each Unicode value is identical for each character, then it will succeed.

Answer 4

perhaps turn off the dataTable unique enforcement constraint, and implement your own unique check method?

Answer 5

That looks like Byte-Order marks (BOM) - http://en.wikipedia.org/wiki/Byte_order_mark

The BOMs are probably being stripped on comparison, hence they'll be the same?

Answer 6

Your issue is probably related to your character being a special unicode character.

"The replacement character � (often a black diamond with a white question mark)
is a symbol found in the Unicode standard at codepoint U+FFFD in the Specials 
table. It is used to indicate problems when a system such as a text parser was
not able to decode a stream of data to a correct symbol"

http://en.wikipedia.org/wiki/Unicode_Specials