Cross platform unicode path handling

Question

I'm using boost::filesystem for cross-platform path manipulation, but this breaks down when calls need to be made down into interfaces I don't control that won't accept UTF-8. For example when using the Windows API, I need to convert to UTF-16, and then call the wide-string version of whatever function I was about to call, and then convert any output back to UTF-8.

While the wpath, and other w* forms of many of the boost::filesystem functions help keep sanity, are there any suggestions for how best to handle this conversion to wide-string forms where needed, while maintaining consistency in my own code?

Answer 1

Well the simplest way would be to make some kind of generic routine that would return string encoded the way you'd want for provided path or a wrapper class around the path.

boost::filesystem::wpath boostPath( L"c:\\some_path" );
MyPathWrapper p( boostPath );
std::wstring sUtf8 = p.file_string_utf8();
std::wstring sUtf16 = p.file_string_utf16();
// or
std::wstring sUtf8 = p.file_string( CP_UTF8 );
// ...

Something like that.

Answer 2

What I did:

struct vpath_traits;

typedef boost::filesystem::basic_path<std::string, vpath_traits> vpath;

// utf8 
struct vpath_traits
{
  typedef std::string internal_string_type;
  typedef std::wstring external_string_type;
  static external_string_type to_external(
      const vpath &, const internal_string_type &src);
  static internal_string_type to_internal(const external_string_type &src);
};


namespace boost {
  namespace filesystem {
    template<> struct is_basic_path<vpath>
    { BOOST_STATIC_CONSTANT( bool, value = true ); };
  }
}

// convenient functions for converting vpath to and from
// whatever API you may be using.

std::string    vpathToWin32Byte(const vpath &src);
vpath          vpathFromWin32Byte(const std::string &str);
vpath          vpathFromWin32Byte(const char *str);

std::wstring   vpathToWin32Wide(const vpath &src);
vpath          vpathFromWin32Wide(const std::wstring &str);
vpath          vpathFromWin32Wide(const wchar_t *str);

QDir           vpathToQDir(const vpath &src);
vpath          vpathFromQDir(const QDir &qdir);

Note:

You must somehow implement the methods vpath_traits::to_external and vpath_traits::to_internal. There are probably some utf8-converter in boost. However I implemented the utf8 to ucs-16 and back myself, but it is neither pretty nor complete, so you are better off using some standard implementation.

It works in our code. E g:

namespace fs = boost::filesystem;
...

vpath dstFile = dstDir / filePath;
bool success = true;

try { fs::remove(dstFile); }
catch (fs::basic_filesystem_error<vpath> & /* e */) { success = false;}

if (success) {
  try { fs::copy_file(srcFile, dstFile); }
  catch (fs::basic_filesystem_error<vpath> & /* e */) { success = false;}
}

...

Answer 3

What works for me:

• Windows: std::wstring is utf16, so "::CreateFileW(Path.file_string().c_str()," ... works
• Posix: you need to convert to utf8, solutions: http://stackoverflow.com/questions/148403/utf8-to-from-wide-char-conversion-in-stl