views:

230

answers:

4

In Python 3.x, a string consists of items of Unicode ordinal. (See the quotation from the language reference below.) What is the internal representation of Unicode string? Is it UTF-16?

The items of a string object are Unicode code units. A Unicode code unit is represented by a string object of one item and can hold either a 16-bit or 32-bit value representing a Unicode ordinal (the maximum value for the ordinal is given in sys.maxunicode, and depends on how Python is configured at compile time). Surrogate pairs may be present in the Unicode object, and will be reported as two separate items.

A: 

It depends: see here. This is still true for Python 3 as far as internal representation goes.

Ned Deily
+1  A: 

I think, Its hard to judge difference between UTF-16, which is just a sequences of 16 bit words, to Python's string object.

And If python is compiled with Unicode=UCS4 option, it will be comparing between UTF-32 and Python string.

So, better consider, they are in different category, although you can transform each others.

S.Mark
+2  A: 

There has been NO CHANGE in Unicode internal representation between Python 2.X and 3.X.

It's definitely NOT UTF-16. UTF-anything is a byte-oriented EXTERNAL representation.

Each code unit (character, surrogate, etc) has been assigned a number from range(0, 2 ** 21). This is called its "ordinal".

Really, the documentation you quoted says it all. Most Python binaries use 16-bit ordinals which restricts you to the Basic Multilingual Plane ("BMP") unless you want to muck about with surrogates (handy if you can't find your hair shirt and your bed of nails is off being de-rusted). For working with the full Unicode repertoire, you'd prefer a "wide build" (32 bits wide).

Briefly, the internal representation in a unicode object is an array of 16-bit unsigned integers, or an array of 32-bit unsigned integers (using only 21 bits).

John Machin
"Storing the unicode codeponts in 16 bit integers" is called "UCS-2". Doing the same thing with 32 bit integers is UCS-4.
Joachim Sauer
I'm not sure how saying that the process is called "UCS2" or "garbelfratzing" or whatever is helping the OP's understanding.
John Machin
calling something by its right name gives you something to label your new understanding with and sort of.. keep it until you encounter again. We can't talk without words.
kaizer.se
+1  A: 

Looking at the source code:

unicodeobject.h:

/* --- Unicode Type ------------------------------------------------------- */

typedef struct {
    PyObject_HEAD
    Py_ssize_t length;   /* Length of raw Unicode data in buffer */
    Py_UNICODE *str;     /* Raw Unicode buffer */
    long hash;    /* Hash value; -1 if not set */
    int state;    /* != 0 if interned. In this case the two
            * references from the dictionary to this object
            * are *not* counted in ob_refcnt. */
    PyObject *defenc;    /* (Default) Encoded version as Python
          string, or NULL; this is used for
          implementing the buffer protocol */
} PyUnicodeObject;

The characters are stored as an array of Py_UNICODE. On most platforms, I believe Py_UNICODE is #defined as wchar_t.

codeape