printing utf8 in glib

Question

Hi.

Why utf8 symbols cannot be printed via glib functions?

Source code:

#include "glib.h"
#include <stdio.h>

int main() {
    g_print("марко\n");
    fprintf(stdout, "марко\n");
}

Build it like this:

gcc main.c -o main $(pkg-config glib-2.0 --cflags --libs)

You could see that glib can't print utf8 and fprintf can:

[marko@marko-work utf8test]$ ./main 
?????
марко

Answer 1

The string passed from g_print() to glibc is not necessarily in UTF-8 encoding since g_print() does character set conversion to the charset specified by the locale.

Answer 2

fprint functions assume that every string you print with them is correctly encoded to match the current encoding of your terminal. g_print() does not assume that and will convert the encoding if it thinks that is necessary; of course this is a bad idea, if the encoding was actually correct before, since that will most likely destroy the encoding. What is the locale setting of your terminal?

You can either set the correct locale by environment variables on most systems or you can do it programatically using the setlocale function. The locale names are system dependent (not part of the POSIX standard), but on most systems the following will work:

#include <locale.h>

:

setlocale(LC_ALL, "en_US.utf8");

Instead of LC_ALL you can also only set the locale for certain operations (e.g. "en_US" will cause English number and date formatting, but maybe you don't want numbers/dates to be formatted that way). To quote from the setlocale man page:

LC_ALL Set the entire locale generically.

LC_COLLATE Set a locale for string collation routines. This controls alphabetic ordering in strcoll() and strxfrm().

LC_CTYPE Set a locale for the ctype(3) and multibyte(3) functions. This controls recognition of upper and lower case, alphabetic or non-alphabetic characters, and so on.

LC_MESSAGES Set a locale for message catalogs, see catopen(3) function.

LC_MONETARY Set a locale for formatting monetary values; this affects the localeconv() function.

LC_NUMERIC Set a locale for formatting numbers. This controls the formatting of decimal points in input and output of floating point numbers in functions such as printf() and scanf(), as well as values returned by localeconv().

LC_TIME Set a locale for formatting dates and times using the strftime() function.

The only two locale values that are always available on all systems are "C", "POSIX" and "".

Only three locales are defined by default: the empty string "" (which denotes the native environment) and the "C" and "POSIX" locales (which denote the C-language environment). A locale argument of NULL causes setlocale() to return the current locale. By default, C programs start in the "C" locale. The only function in the library that sets the locale is setlocale(); the locale is never changed as a side effect of some other routine.

Answer 3

Usually it is not recommended to use anything other than ASCII inside text files. You should use tools like gettext in order to translate words from different languages. If this is out of the question then you should store your string in UTF-8 in your code.

Try printing this one (it's the hexadecimal representation of your string):

char hex_marco[]={0xD0, 0xBC, 0xD0, 0xB0, 0xD1, 0x80, 0xD0, 0xBA, 0xD0, 0xBE, 0};

This works for me in printf (cannot test here with glib):

#include <stdio.h>

char hex_marco[]={0xD0, 0xBC, 0xD0, 0xB0, 0xD1, 0x80, 0xD0, 0xBA, 0xD0, 0xBE, 0};

int main(void)
{
    printf("%s\n",hex_marco);
    return 0;
}

Redirect the output to file and see it as UTF-8.

Hope it helps.

Answer 4

You need to initialize the locale's encoding by calling setlocale at your program's start.

setlocale(LC_CTYPE, "")

This is normally carried out for you if you use some initialization function like gtk_init(..) or similar.