I found this in the wikipedia article on utf-8:
Sorting of UTF-8 strings as arrays of unsigned bytes will produce the same results as sorting them based on Unicode code points.
That would lead me to believe that for comparison purposes (sorting, binary search, etc) that comparing two byte arrays (i.e. byte-by-byte like memcmp) of utf-8 encoded strings would give the same results as comparing the actual unicode strings.
Is this true?
No, it isn't.
For example, À can be written as a single codepoint (U+00C0 LATIN CAPITAL LETTER A WITH GRAVE) or as two codepoints (U+0041 LATIN CAPITAL LETTER A U+0300 COMBINING GRAVE ACCENT).
Both representations should compare equal, but will have different byte encodings.
Yes, given that there's a one-to-one mapping between sequences bytes in UTF-8 encoding and Unicode code points.
However, there are way to compare Unicode strings besides looking at the raw code points. If you just look at code points -- or UTF-8 bytes -- as numbers then you miss culture-specific comparison logic.
To implement comparison and sorting correctly for a specific culture, on .NET, you should use the standard string comparison functions.
It depends on what you mean by "comparing the actual Unicode strings".
If you're just going to compare the code points (as 32-bit numbers) instead of the UTF-8 encoded code points, then the answer is yes: that will give the same results. The mapping from code points to UTF-8 encoded bytes is one-to-one.
If you're going to do a proper Unicode string comparison, instead of bytewise comparison of the UTF-8, the answer is no. In Unicode, there can be different ways to represent the same character. For example, é can be represented in (at least) two ways:
U+00e9 (LATIN SMALL LETTER E WITH ACUTE), orU+0065 (LATIN SMALL LETTER E) followed by U+0301 (COMBINING ACUTE ACCENT).A properly written Unicode comparison function will consider these two to be identical.
I found this in the wikipedia article on utf-8:
Sorting of UTF-8 strings as arrays of unsigned bytes will produce the same results as sorting them based on Unicode code points.
That would lead me to believe that for comparison purposes (sorting, binary search, etc) that comparing two byte arrays (i.e. byte-by-byte like memcmp) of utf-8 encoded strings would give the same results as comparing the actual unicode strings.
This all depends on what you mean by "actual Unicode strings" and what you mean by "comparing". In the .Net Framework strings are in the UTF-16 form of Unicode. A simple binary comparison between UTF-16 strings will yeild a different sort order than the same comparison between UTF-8 and UTF-32 (the code-point version referenced in the quote) strings.
But a binary comparison of any of those things is not very useful. You should be using the built-in culture-aware comparisons. This is because two string that are, for all intents and purposes, the same can be constructed from different sequences of code points. The built-in comparisons take those things into account.
It is the same as a code-point for code-point comparison, that is to say one that pays no attention to case-folding, cultural orderings, composition, or anything other than the Unicode value.
This is pretty useless when considering strings as a piece of human-readable text, but sometimes you just want to be able to put the strings into an ordering, as some algorithms (binary search as you say) need a consistent ordering, but the details of that consistent ordering is not significant.
It is important to note though, that the ordinal comparison on strings offered by .NET works on the UTF-16 used internally which does not maintain code-point ordering. If we compare a string with just the character U+FF61 and a string with just the character U+10002, then .NET will store the latter as surrogate pairs, of 0xD800 and 0XDC02.
Hence:
string.CompareOrdinal("\U0000ff61", "\U00010002");
and
string.Compare("\U0000ff61", "\U00010002", StringComparison.Ordinal);
both return values great than zero, even though the former is lower in code-point value than the latter (I used the \U form rather than the \u form to make that clearer).
If by "the actual unicode strings" you mean the .NET UTF-16 strings, then the answer to your question is no, for the opposite reason to that which led to your thinking it might work.