Truncating unicode so it fits a maximum size when encoded for wire transfer

Answer 1

For JSON formatting (unicode escape, e.g. \uabcd), I am using the following algorithm to achieve this:

• Encode the Unicode string into the backslash-escape format which it would eventually be in the JSON version
• Truncate 3 bytes more than my final limit
• Use a regular expression to detect and chop off a partial encoding of a Unicode value

So (in Python 2.5), with some_string and a requirement to cut to around 100 bytes:

# Given some_string is a long string with arbitrary Unicode data.
encoded_string = some_string.encode('unicode_escape')
partial_string = re.sub(r'([^\\])\\(u|$)[0-9a-f]{0,3}$', r'\1', encoded_string[:103])
final_string   = partial_string.decode('unicode_escape')

Now final_string is back in Unicode but guaranteed to fit within the JSON packet later. I truncated to 103 because a purely-Unicode message would be 102 bytes encoded.

Disclaimer: Only tested on the Basic Multilingual Plane. Yeah yeah, I know.

Answer 2

One of UTF-8's properties is that it is easy to resync, that is find the unicode character boundaries easily in the encoded bytestream. All you need to do is to cut the encoded string at max length, then walk backwards from the end removing any bytes that are > 127 -- those are part of, or the start of a multibyte character.

As written now, this is too simple -- will erase to last ASCII char, possibly the whole string. What we need to do is check for no truncated two-byte (start with 110yyyxx) three-byte (1110yyyy) or four-byte (11110zzz)

Python 2.6 implementation in clear code. Optimization should not be an issue -- regardless of length, we only check the last 1-4 bytes.

# coding: UTF-8

def decodeok(bytestr):
    try:
        bytestr.decode("UTF-8")
    except UnicodeDecodeError:
        return False
    return True

def is_first_byte(byte):
    """return if the UTF-8 @byte is the first byte of an encoded character"""
    o = ord(byte)
    return ((0b10111111 & o) != o)

def truncate_utf8(bytestr, maxlen):
    u"""

    >>> us = u"ウィキペディアにようこそ"
    >>> s = us.encode("UTF-8")

    >>> trunc20 = truncate_utf8(s, 20)
    >>> print trunc20.decode("UTF-8")
    ウィキペディ
    >>> len(trunc20)
    18

    >>> trunc21 = truncate_utf8(s, 21)
    >>> print trunc21.decode("UTF-8")
    ウィキペディア
    >>> len(trunc21)
    21
    """
    L = maxlen
    for x in xrange(1, 5):
        if is_first_byte(bytestr[L-x]) and not decodeok(bytestr[L-x:L]):
            return bytestr[:L-x]
    return bytestr[:L]

if __name__ == '__main__':
    # unicode doctest hack
    import sys
    reload(sys)
    sys.setdefaultencoding("UTF-8")
    import doctest
    doctest.testmod()

Answer 3

Hi,

This will do for UTF8, If you like to do it in regex.

import re

partial="\xc2\x80\xc2\x80\xc2"

re.sub("([\xf6-\xf7][\x80-\xbf]{0,2}|[\xe0-\xef][\x80-\xbf]{0,1}|[\xc0-\xdf])$","",partial)

"\xc2\x80\xc2\x80"

Its cover from U+0080 (2 bytes) to U+10FFFF (4 bytes) utf8 strings

Its really straight forward just like UTF8 algorithm

From U+0080 to U+07FF It will need 2 bytes 110yyyxx 10xxxxxx Its mean, if you see only one byte in the end like 110yyyxx (0b11000000 to 0b11011111) It is [\xc0-\xdf], it will be partial one.

From U+0800 to U+FFFF is 3 bytes needed 1110yyyy 10yyyyxx 10xxxxxx If you see only 1 or 2 bytes in the end, it will be partial one. It will match with this pattern [\xe0-\xef][\x80-\xbf]{0,1}

From U+10000–U+10FFFF is 4 bytes needed 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx If you see only 1 to 3 bytes in the end, it will be partial one It will match with this pattern [\xf6-\xf7][\x80-\xbf]{0,2}

Update :

If you only need Basic Multilingual Plane, You can drop last Pattern. This will do.

re.sub("([\xe0-\xef][\x80-\xbf]{0,1}|[\xc0-\xdf])$","",partial)

Let me know if there is any problem with that regex.

Answer 4

def unicode_truncate(s, length, encoding='utf-8'):
    encoded = s.encode(encoding)[:length]
    return encoded.decode(encoding, 'ignore')

Here is an example for unicode string where each character is represented with 2 bytes in UTF-8:

>>> unicode_truncate(u'абвгд', 5)
u'\u0430\u0431'