Convert utf-8 std::string to std::wstring on iPhone

Question

Hi, I have a UTF-8 string (created an std::string from a byte array) I understand that the encoding means that the size()/length() won't give me the actual number of glyphs if the text is chinese for instance... I understand that in order to get the unicode character code of each glyph I need to convert it to wstring (or any UTF>8 representation) and then I can get the value that will represent what I want.

I've looked around and haven't found any simple way to do it with std c++. What am I missing?

I'm compiling gcc 4+ on Apple's iPhone using cocoa-touch framework.

Answer 1

There is no notion of utf-8 or unicode in the C++ standard. You should check your available APIs or an external libraries to perform your conversions.

Or you can do yourself the function to check the real number of characters from a utf-8 encoded std::string, I think it's not that difficult if you know how utf-8 works.

Answer 2

To get the number of utf8 'characters/code points' in a std::string you could do this : Traverse the string, if the char is between 0 and 127, it's a one byte character, between 194 and 223 it's a 2 bytes character (so advance in consequence), between 224 and 239 it's a 3 bytes character (so advance in consequence), between 240 and 244 it's a 4 bytes character (so advance in consequence).

Since wchar_t on the Iphone is, I guess, 32bits, if you really want a wstring you could use UTF8CPP to convert to UTF32. UTF8CPP could also give you the code points of your string.

But I don't understand why you're using C++ for the Iphone ? Look here : Objective-C Tuesdays: wide character strings

Answer 3

Boost provides a UTF-8 codecvt facet. You should be able to invoke it directly to perform conversions between UTF-8 encoded bytes and 32-bit wchar_t.

Answer 4

Well it is not simple and I have not used it myself but the locale classes should help with converting your string. From the description you can use the ctype::widen method ot convert between a char and a wchar.

Answer 5

First of all, even if you convert your UTF-8 string to UTF-32 (and store it in wstring) it does not mean each wchar_t will correspond to a single glyph. See this text for some of the issues: http://www.unicode.org/reports/tr15/ .

Having said that, if you really need to convert a UTF-8 encoded string to UTF-32, you can use UTF-8 CPP library like this:

wstring utf32result;
utf8::utf8to32(utf8string.begin(), utf8string.end(), back_inserter(utf32result));