Using poll on file-like object returned by urllib2.urlopen()?

Question

I've run into the bug described at http://bugs.python.org/issue1327971 while trying to poll a file-like object returned by urllib2.urlopen().

Unfortunately, being relatively new to Python, I can't actually determine from the responses how to get around the issue as they seem mostly geared towards fixing the bug, rather than hacking the code that triggers it to work.

Here is a distilled version of my code that throws the error:

import urllib2, select

if __name__ == "__main__":
    p = select.poll()
    url = "http://localhost/"
    fd = urllib2.urlopen(url)
    p.register(fd, select.POLLIN | select.POLLERR | select.POLLHUP | select.POLLNVAL)

    result = p.poll()
    for fd, event in result:
        if event == select.POLLIN:
            while 1:
                buf = fd.read(4096)
                if not buf:
                    break
                print buf

And the error which is raised when I run it on python 2.6:

Traceback (most recent call last):
  File "/home/shab/py/test.py", line 9, in <module>
    p.register(fd, select.POLLIN | select.POLLERR | select.POLLHUP | select.POLLNVAL)
  File "/usr/lib/python2.6/socket.py", line 287, in fileno
    return self._sock.fileno()
AttributeError: HTTPResponse instance has no attribute 'fileno'

Update: I do not want to modify the system libraries.

Answer 1

It looks like you want to modify urllib with this patch. Keep in mind, there's a reason this code hasn't been released. It hasn't been completely reviewed.

EDIT: Actually, I think you want to modify httplib with the other patch.

Answer 2

If you don't want to modify you system libraries you also can patch httplib on the fly to match the patch in the bug report:

import httplib

@property
def http_fileno(self):
   return self.fp.fileno

@http_fileno.setter
def http_fileno(self, value):
   self.fp.fileno = value

httplib.HTTPResponse.fileno = http_fileno

# and now on with the previous code
# ...

You then get an error on fd.read(4096) because the fd returned by poll is a raw file descriptor value, not a file-like object. You probably need to use the original file object to read the data, not the value returned by poll.