How do I pass compressed data using the Task Queue Python API in App Engine?

Question

I'm trying to use compressed data with my Tasks in the Task Queue like so:

t = taskqueue.Task(url='/tasks/queue',
                   params={'param': zlib.compress(some_string)}

However when I try to decompress it in the queue handler like so

message = self.request.get('param')
message = zlib.decompress(message)

I get this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u06b8' in position 2: ordinal not in range(128)

Anyone know of what's going on here? Is there a work around?

Answer 1

Read the docs... (my emphasis!):

params Dictionary of parameters to use for this Task. Values in the dictionary may be iterable to indicate repeated parameters. May not be specified for a POST request if payload is already specified. For POST requests, these params will be encoded as 'application/x-www-form-urlencoded' and set to the payload; for all other methods, the parameters will be converted to a query string. May not be specified if the URL already contains a query string and the method is GET.

zlib.compress produces an arbitrary string of bytes... but then query-string conversion interprets it as Unicode! So, use any 1-byte codec, such as latin-1, to .encode the compressed results in order to pass (what's actually a binary) bytestring of params, and the same codec for a .decode to get back from the "unicode" string to a string of bytes that you can decompress. Phew... you sure the compression is crucial enough to your app's performance to be worth this weird set of gyrations, or wouldn't it be better to eschew it?-)

Answer 2

Instead of using params, use payload, which includes your data in the body of the request, unencoded. Then you can use zlib.decompress(self.request.body) to retrieve the data.