How to deal with Unicode strings in C/C++ in a cross-platform friendly way?

Question

On platforms different than Windows you could easily use char * strings and treat them as UTF-8.

The problem is that on Windows you are required to accept and send messages using wchar* strings (W). If you'll use the ANSI functions (A) you will not support Unicode.

So if you want to write truly portable application you need to compile it as Unicode on Windows.

Now, In order to keep the code clean I would like to see what is the recommended way of dealing with strings, a way that minimize ugliness in the code.

Type of strings you may need: std::string, std::wstring, std::tstring,char *,wchat_t *, TCHAR*, CString (ATL one).

Issues you may encounter:

• cout/cerr/cin and their Unicode variants wcout,wcerr,wcin
• all renamed wide string functions and their TCHAR macros - like strcmp, wcscmp and _tcscmp.
• constant strings inside code, with TCHAR you will have to fill your code with _T() macros.

What approach do you see as being best? (examples are welcome)

Personally I would go for a std::tstring approach but I would like to see how would do to the conversions where they are necessary.

Answer 1

I can only suggest you to check this library out: http://cppcms.sourceforge.net/boost_locale/docs/
It might help, it's a boost candidate for now but I believe it will make it.

Answer 2

You can keep all your strings UTF-8 encoded and just convert them to UTF-16 before interacting with WIn32 API. Take a look at UTF8-CPP library for some easy to use conversion functions

Answer 3

If you writing portable code:

1st Never use wchar_t it is nor portable and its encoding is not well defined between platforms (utf-16 windows/utf-32 all others).

Never use TChar, use plain std::string encoded as UTF-8.

When dealing with Brain Damaged Win32 API just convert UTF-8 string to UTF-16 before calling it.

See http://stackoverflow.com/questions/1049947/should-utf-16-be-considered-harmful as well about how Windows project adopt UTF-8 as main encoding.