HTTP POST binary files using Python: concise non-pycurl examples?

Question

Hi,

I'm interested in writing a short python script which uploads a short binary file (.wav/.raw audio) via a POST request to a remote server.

I've done this with pycurl, which makes it very simple and results in a concise script; unfortunately it also requires that the end user have pycurl installed, which I can't rely on.

I've also seen some examples in other posts which rely only on basic libraries, urllib, urllib2, etc., however these generally seem to be quite verbose, which is also something I'd like to avoid.

I'm wondering if there are any concise examples which do not require the use of external libraries, and which will be quick and easy for 3rd parties to understand - even if they aren't particularly familiar with python.

What I'm using at present looks like,

def upload_wav( wavfile, url=None, **kwargs ):
    """Upload a wav file to the server, return the response."""

    class responseCallback:
        """Store the server response."""
        def __init__(self):
            self.contents=''
        def body_callback(self, buf):
            self.contents = self.contents + buf

        def decode( self ):
            self.contents = urllib.unquote(self.contents)
            try:
                self.contents = simplejson.loads(self.contents)
            except:
                return self.contents

    t = responseCallback()
    c = pycurl.Curl()
    c.setopt(c.POST,1)
    c.setopt(c.WRITEFUNCTION, t.body_callback)
    c.setopt(c.URL,url)
    postdict = [
        ('userfile',(c.FORM_FILE,wavfile)),  #wav file to post                                                                                 
        ]
    #If there are extra keyword args add them to the postdict                                                                                  
    for key in kwargs:
        postdict.append( (key,kwargs[key]) )
    c.setopt(c.HTTPPOST,postdict)
    c.setopt(c.VERBOSE,verbose)
    c.perform()
    c.close()
    t.decode()
    return t.contents

this isn't exact, but it gives you the general idea. It works great, it's simple for 3rd parties to understand, but it requires pycurl.

Answer 1

How's urllib substantially more verbose? You build postdict basically the same way, except you start with

postdict = [ ('userfile', open(wavfile, 'rb').read()) ]

Once you vave postdict,

resp = urllib.urlopen(url, urllib.urlencode(postdict))

and then you get and save resp.read() and maybe unquote and try JSON-loading if needed. Seems like it would be actually shorter! So what am I missing...?

Answer 2

POSTing a file requires multipart/form-data encoding and, as far as I know, there's no easy way (i.e. one-liner or something) to do this with the stdlib. But as you mentioned, there are plenty of recipes out there.

Although they seem verbose, your use case suggests that you can probably just encapsulate it once into a function or class and not worry too much, right? Take a look at the recipe on ActiveState and read the comments for suggestions:

• Recipe 146306: Http client to POST using multipart/form-data

or see the MultiPartForm class in this PyMOTW, which seems pretty reusable:

• PyMOTW: urllib2 - Library for opening URLs.

I believe both handle binary files.

Answer 3

urllib.urlencode doesn't like some kinds of binary data.