Detecting locale from unicode string in c++

Question

I have a string and I want to check if the content is in English or Hindi(My local language). I figured out that the unicode range for hindi character is from U0900-U097F.

What is the simplest way to find if the string has any characters in this range?

I can use std::string or Glib::ustring depending on whichever is convenient.

Answer 1

The first step is writing a functor to tell if a given wchar_t is Hindi. This will be (derived from) a std::unary_function<wchar_t, bool>. Implementation is trivial: return c>= 0x0900 && c < 0x980;. The second step is using it: std::find_if(begin, end, is_hindi()).

Since you'll need Unicode, you should probably use wchar_t and therefore std::wstring. Neither std::string nor GLib::ustring supports Unicode proper. On some systems (Windows in particular) the implementation of wchar_t is restricted to Unicode 4 = 16 bits but that should still be enough for 99.9% of the worlds population.

You'll need to convert from/to UTF-8 on I/O, but the advantage of "one character = one wchar_t" is big. For instance, std::wstring::substr() will work reasonably. You might still have issues with "characters" like U+094B (DEVANAGARI VOWEL SIGN O), though. When iterating over a std::wstring, that will appear to be a character by itself, instead of a modifier. That's still better than std::string with UTF-8, where you'd end up iterating over the individual bytes of U+094B. And to take just your original examples, none of the bytes in UTF8(U+094B) are reserved for Hindi.

Answer 2

If the string is already encoded as UTF-8, I would not convert it to UTF-16 (I assume that's what MSalters calls "Unicode proper") but iterate through the UTF-8 encoded string and check whether there is a Hindi character in it.

With std::string, you can easily iterate with the help of the UTF8-CPP library: - take a look at utf8::next() function, or the iterator class.

GLib::ustring has an iterator that seems to enable the same functionality (haven't tried it):

Answer 3

Here is how you do it with Glib::ustring :

using Glib::ustring;

ustring x("सहस");    // hindi string
bool is_hindi = false;
for (ustring::iterator i = x.begin(); i != x.end(); i ++)
    if (*i >= 0x0900 && *i <= 0x097f)
        is_hindi = true;