Visual C++: Migrating traditional C and C++ string code to a Unicode world

Question

I see that Visual Studio 2008 and later now start off a new solution with the Character Set set to Unicode. My old C++ code deals with only English ASCII text and is full of:

• Literal strings like "Hello World"
• char type
• char * pointers to allocated C strings
• STL string type

• Conversions from STL string to C string and vice versa using STL string constructor (which accepts const char *) and STL string.c_str()

  1. What are the changes I need to make to migrate this code so that it works in an ecosystem of Visual Studio Unicode and Unicode enabled libraries? (I have no real need for it work with both ASCII and Unicode, it can be pure Unicode.)

  2. Is it also possible to do this in a platform independent way? (i.e., by not using Microsoft types.)

I see so many wide character and Unicode types and conversions scattered around, hence my confusion. (Ex: wchar_t, TCHAR, _T, _TEXT, TEXT etc.)

Answer 1

• Around your literal constants with _T(), e.g. _T("Hello world")
  
• Replace char with macros CHAR
• Replace string with wstring

Then all should work.

Answer 2

"Hello World" -> L"Hello World"

char -> wchar_t (unless you actually want char)

char * -> wchar_t *

string -> wstring

These are all platform independent. However, be aware that a wide character may be different on different platforms (two bytes on windows, four bytes on others).

Define UNICODE and _UNICODE in your project (in Visual Studio you can do this by setting the project to use Unicode in the settings). This also makes _T, TCHAR, _TEXT and TEXT macros to become L automatically. These are Microsoft specific, so avoid these if you want to be cross-platform.

Answer 3

I would suggest not to worry about supporting both ascii and unicode build (a-la TCHAR) and go stright to unicode. That way you get to use more of the platform independant functions (wcscpy, wcsstr etc) instead of relying onto TCHAR functions which are Micrpsoft specific.

You can use std::wstring instead of std::string and replace all chars with wchar_ts. With a massive change like this I found that you start with one thing and let the compiler guide you to the next.

One thing that I can think of that might not be obvious at run time is where a string is allocated with malloc without using sizeof operator for the underlying type. So watch out for things like char * p = (char*)malloc(11) - 10 characters plus terminating NULL, this string will be half the size it's supposed to be in wchar_ts. It should become wchar_t * p = (wchar_t*)malloc(11*sizeof(wchar_t)).

Oh and the whole TCHAR is to support compile time ASCII/Unicode strings. It's defined something like this:

#ifdef _UNICODE
#define _T(x) L ## x
#else
#define _T(x) ## x
#endif

So that in unicode configuration _T("blah") becomes L"blah" and in ascii configuration it's "blah".

Answer 4

I recommend very much against L"", _T(), std::wstring (the latter is not multiplatform) and Microsoft recommendations on how to do Unicode.

There's a lot of confusion on this subject. Some people still think Unicode == 2 byte characters == UTF-16. Neither equality is correct.

In fact, it's possible, and even better to stay with char* and the plain std::string, plain literals and change very little (and still fully support Unicode!).

See my answer here: http://stackoverflow.com/questions/1049947/should-utf-16-be-considered-harmful/1855375#1855375 for how to do it the easiest (in my opinion) way.

Answer 5

Your question involves two different but related concepts. One of them is the encoding of the string (Unicode/ASCII, for example). The other is the data type to be used for the character representation.

Technically, you can have an Unicode application using plain char and std::string. You could use literals in hexadecimal ("\x5FA") or octal ("\05FA") format to specify the byte sequence of the string. Notice that with this approach your already existent string literals that contain ASCII characters should remain valid, since Unicode preserves the codes from ASCII.

One important point to observe is that many string related functions would need to be used carefully. This is because they'll be operating on bytes rather than characters. For example, std::string::operator[] might give you a particular byte that is only part of an Unicode character.

In Visual Studio wchar_t was chosen as the underlying character type. So if you're in working with Microsoft based libraries things should get easier for you if you follow many of the advices posted by others here. Replacing char for wchar_t, using the "T" macros (if you want to preserve transparency between Unicode/non-Unicode), etc.

However, I don't think there is a de facto standard of working with Unicode across libraries, since they might have different strategies to handle it.