Python interface to PayPal - urllib.urlencode non-ASCII characters failing

Question

I am trying to implement PayPal IPN functionality. The basic protocol is as such:

1. The client is redirected from my site to PayPal's site to complete payment. He logs into his account, authorizes payment.
2. PayPal calls a page on my server passing in details as POST. Details include a person's name, address, and payment info etc.
3. I need to call a URL on PayPal's site internally from my processing page passing back all the params that were passed in abovem and an additional one called 'cmd' with a value of '_notify-validate'.

When I try to urllib.urlencode the params which PayPal has sent to me, I get a:

While calling send_response_to_paypal. Traceback (most recent call last):
  File "<snip>/account/paypal/views.py", line 108, in process_paypal_ipn
    verify_result = send_response_to_paypal(params)
  File "<snip>/account/paypal/views.py", line 41, in send_response_to_paypal
    params = urllib.urlencode(params)
  File "/usr/local/lib/python2.6/urllib.py", line 1261, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 9: ordinal not in range(128)

I understand that urlencode does ASCII encoding, and in certain cases, a user's contact info can contain non-ASCII characters. This is understandable. My question is, how do I encode non-ASCII characters for POSTing to a URL using urllib2.urlopen(req) (or other method)

Details:

I read the params in PayPal's original request as follows (the GET is for testing):

def read_ipn_params(request):
    if request.POST:  
        params= request.POST.copy()  
        if "ipn_auth" in request.GET:
            params["ipn_auth"]=request.GET["ipn_auth"]
        return params
    else:  
        return request.GET.copy()

The code I use for sending back the request to PayPal from the processing page is:

def send_response_to_paypal(params):
    params['cmd']='_notify-validate'  
    params = urllib.urlencode(params)  
    req = urllib2.Request(PAYPAL_API_WEBSITE, params)  
    req.add_header("Content-type", "application/x-www-form-urlencoded") 
    response = urllib2.urlopen(req)  
    status = response.read()  
    if not status == "VERIFIED":  
        logging.warn("PayPal cannot verify IPN responses: " + status)
        return False

    return True

Obviously, the problem only arises if someone's name or address or other field used for the PayPal payment does not fall into the ASCII range.

Answer 1

Try converting the params dictionary to utf-8 first... urlencode seems to like that better than unicode:

params = urllib.urlencode(dict([k, v.encode('utf-8')] for k, v in params.items()))

Of course, this assumes your input is unicode. If your input is something other than unicode, you'll want to decode it to unicode first, then encode it:

params['foo'] = my_raw_input.decode('iso-8859-1')
params = urllib.urlencode(dict([k, v.encode('utf-8')] for k, v in params.items()))

Answer 2

Instead of encoding to utf-8, one should encode to what ever the paypal is using for the post. It is available under key 'charset' in the form paypal sends.

So the following worked for me:

data = dict([(k, v.encode(data['charset'])) for k, v in data.items()])

Answer 3

I know it's a bit late to chime in here, but the best solution I found was to not even parse what they were giving back. In django (don't know what you're using) I was able to get the raw request they sent, which I passed back verbatim. Then it was just a matter of putting the cmd key onto that.

This way it never matters what encoding they send you, you're just sending it right back.