Python stream http client with keep-alive

Question

Hi.

I need a python http client that can reuse connections and that supports consuming the stream as it comes in. It will be used to parse xml streams, sax style.

I came up with a solution, but I'm not sure it is the best one (there are quite a few ways of writing an http client in python)

class Downloader():

    def __init__(self, host):
            self.conn = httplib.HTTPConnection(host)

    def get(self, url):
            self.conn.request("GET", url)
            resp = self.conn.getresponse()
            while True:
                    data = resp.read(10)
                    if not data:
                            break
                    yield data

Thanks folks!

Answer 1

urlgrabber supports keepalive and can return a file-like.

Answer 2

There is also pycurl. By default keepalive is turned on and you can write to a file for output.

Follow the examples, they are quite helpful