Unescape/unquote binary strings in (extended) url encoding in python

Question

Hi,

for analysis I'd have to unescape URL-encoded binary strings (non-printable characters most likely). The strings sadly come in the extended URL-encoding form, e.g. "%u616f". I want to store them in a file that then contains the raw binary values, eg. 0x61 0x6f here.

How do I get this into binary data in python? (urllib.unquote only handles the "%HH"-form)

Answer 1

I guess you will have to write the decoder function by yourself. Here is an implementation to get you started:

def decode(file):
    while True:
        c = file.read(1)
        if c == "":
            # End of file
            break
        if c != "%":
            # Not an escape sequence
            yield c
            continue
        c = file.read(1)
        if c != "u":
            # One hex-byte
            yield chr(int(c + file.read(1), 16))
            continue
        # Two hex-bytes
        yield chr(int(file.read(2), 16))
        yield chr(int(file.read(2), 16))

Usage:

input = open("/path/to/input-file", "r")
output = open("/path/to/output-file", "wb")
output.writelines(decode(input))
output.close()
input.close()

Answer 2

Here is a regex-based approach:

# the replace function concatenates the two matches after 
# converting them from hex to ascii
repfunc = lambda m: chr(int(m.group(1), 16))+chr(int(m.group(2), 16))

# the last parameter is the text you want to convert
result = re.sub('%u(..)(..)', repfunc, '%u616f')
print result

gives

ao

Answer 3

The strings sadly come in the extended URL-encoding form, e.g. "%u616f"

Incidentally that's not anything to do with URL-encoding. It's an arbitrary made-up format produced by the JavaScript escape() function and pretty much nothing else. If you can, the best thing to do would be to change the JavaScript to use the encodeURIComponent function instead. This will give you a proper, standard URL-encoded UTF-8 string.

e.g. "%u616f". I want to store them in a file that then contains the raw binary values, eg. 0x61 0x6f here.

Are you sure 0x61 0x6f (the letters "ao") is the byte stream you want to store? That would imply UTF-16BE encoding; are you treating all your strings that way?

Normally you'd want to turn the input into Unicode then write it out using an appropriate encoding, such as UTF-8 or UTF-16LE. Here's a quick way of doing it, relying on the hack of making Python read '%u1234' as the string-escaped format u'\u1234':

>>> ex= 'hello %e9 %u616f'
>>> ex.replace('%u', r'\u').replace('%', r'\x').decode('unicode-escape')
u'hello \xe9 \u616f'

>>> print _
hello é 慯

>>> _.encode('utf-8')
'hello \xc2\xa0 \xe6\x85\xaf'