C++ unicode question

Question

I'm aware of ICU and small libraries like the utf8 one on code project (forget the exact name) however none of these are exactly what I want.

What I really want is something like ICU but wrapped up in a more friendly manner.

Specifically:

• Fully Object Orientated
• Implementations of the c++ standard streams, or at least something that performs the same role.
• Can format time, dates etc in a locale dependent manner (eg dd/mm/yy in the UK and mm/dd/yy in the US).
• Lets me choose the "internal" encoding of strings, so I can for example make it use UTF-16 on windows to avoid lots of conversions when passing strings to and from the windows API and DirectX
• Easy converting of strings between encodings

If no such library exists, is it possible to wrap the ICU up using the standard c++ classes, so I can for example create a ustring which has identical usage to std::string and std::wstring, and also implement versions of the streams (optimally with them being fully compatible with the existing ones, ie I could pass it to a function expecting an std::ostream and it will perform conversion between its internal format and ascii (or utf-8) on the fly)? Assuming it is possible just how much work would it be?

EDIT: Also having looked at the c++0x standard and noticed literals for utf8, utf16 and utf32, does that mean that standard library (eg strings, streams, etc) will fully support those encodeings and the conversion between them? If so anyone got any idea how long it will be until Visual Studio will support those features?

EDIT2: As for using the existing c++ support, I'll look up the locale and facet stuff.

One of the problems I ran into is that when using streams defined around wchar_t which is 2 bytes under windows for file i/o however is it still seemed to use ascii for the files them selves.

std::wofstream file(L"myfile.txt", std::ios::out);
file << L"Hello World!" << std::endl;

resulted in the following hex in the file
48 65 6C 6C 6F 20 57 6F 72 6C 64 0D 0A
which is clearly ascii rather than the expected utf-16 output of:
FF FE 48 00 65 00 6C 00 6C 00 6F 00 20 00 57 00 6F 00 72 00 6C 00 64 00 0D 00 0A 00

Answer 1

Formatting date, time etc can be done by specifying a particular locale. As for rolling your own -- it is always possible, taking as much or as little from the underlying library as you need.

Also having looked at the c++0x standard and noticed literals for utf8, utf16 and utf32, does that mean that standard library (eg strings, streams, etc) will fully support those encodeings and the conversion between them?

Yes. But note these are different data types and not your regular wchar sequence or a wstring.

If so anyone got any idea how long it will be until Visual Studio will support those features?

To the best of my knowledge: vc9 (VS2008) only has partial support for some TR1 features. vc10 (VS2010) is expected to have a better support.

Answer 2

What I really want is something like ICU but wrapped up in a more friendly manner

Unfortunatly, there is no such thing. Their API is not SO terrible, so you can get used to it for some effort.

Can format time, dates etc in a locale dependent manner (eg dd/mm/yy in the UK and mm/dd/yy in the US).

There is a full support of it in std::locale class, read on how to use it. You can also specify locale for std::iostream so it would format numers, dates correctly.

Easy converting of strings between encodings

std::locale provides facets for coverting 8bits local encoding to wide one and back.

so I can for example make it use UTF-16

ICU uses utf-16 internally, win32 wchar_t and wstring use utf-16 as well, under other OSes most of implementations give wchar_t as utf-32 and wstring uses utf-32.

Remarks: Support of std::locale is not perfect, but it already gives many tools that are useful for charrecter manipulations.

See: http://www.cplusplus.com/reference/std/locale/

Answer 3

I did my own small wrapper. I can share if you want.

Answer 4

Tough luck. I know that Dinkumware libraries offer some Unicode support - you may look at the documentation at their web site. AFAIK, it is not free.

Answer 5

This is how I use ICU to convert between std::string (in UTF-8) and std::wstring

/** Converts a std::wstring into a std::string with UTF-8 encoding.
 */
template < typename StringT >
StringT utf8 ( std::wstring const & rc_string );

/** Converts a std::String with UTF-8 encoding into a std::wstring.
 */
template < typename StringT >
StringT utf8 ( std::string const & rc_string );

/** Nop specialization for std::string.
 */
template < >
inline std::string utf8 ( std::string const & rc_string )
{
  return rc_string;
}

/** Nop specialization for std::wstring.
 */
template < >
inline std::wstring utf8 ( std::wstring const & rc_string )
{
  return rc_string;
}

template < >
std::string utf8 ( std::wstring const & rc_string )
{
  std::string result;
  if(rc_string.empty())
    return result;

  std::vector<UChar> buffer;

  result.resize(rc_string.size() * 3); // UTF-8 uses max 3 bytes per char
  buffer.resize(rc_string.size() * 2); // UTF-16 uses max 2 bytes per char

  UErrorCode status = U_ZERO_ERROR;
  int32_t len = 0;

  u_strFromWCS(
    &buffer[0],
    buffer.size(),
    &len,
    &rc_string[0],
    rc_string.size(),
    &status
  );
  if(!U_SUCCESS(status))
  {
    throw XXXException("utf8: u_strFromWCS failed");
  }
  buffer.resize(len);

  u_strToUTF8(
    &result[0],
    result.size(),
    &len,
    &buffer[0],
    buffer.size(),
    &status
  );
  if(!U_SUCCESS(status))
  {
    throw XXXException("utf8: u_strToUTF8 failed");
  }
  result.resize(len);

  return result;
}/* end of utf8 ( ) */


template < >
std::wstring utf8 ( std::string const & rc_string )
{
  std::wstring result;
  if(rc_string.empty())
    return result;

  std::vector<UChar> buffer;

  result.resize(rc_string.size());
  buffer.resize(rc_string.size());

  UErrorCode status = U_ZERO_ERROR;
  int32_t len = 0;

  u_strFromUTF8(
    &buffer[0],
    buffer.size(),
    &len,
    &rc_string[0],
    rc_string.size(),
    &status
  );
  if(!U_SUCCESS(status))
  {
    throw XXXException("utf8: u_strFromUTF8 failed");
  }
  buffer.resize(len);

  u_strToWCS(
    &result[0],
    result.size(),
    &len,
    &buffer[0],
    buffer.size(),
    &status
  );
  if(!U_SUCCESS(status))
  {
    throw XXXException("utf8: u_strToWCS failed");
  }
  result.resize(len);

  return result;
}/* end of utf8 ( ) */

Using it is as simple as that:

std::string s = utf8<std::string>(std::wstring(L"some string"));
std::wstring s = utf8<std::wstring>(std::string("some string"));