reading a stream made by urllib2 never recovers when connection got interrupted

Answer 1

Good question, I would be really interested in finding an answer. The only workaround I could think of is using the signal trick explained in python docs. In your case it will be more like:

import signal
import urllib2

def read(url):
    stream = urllib2.urlopen(url)
    return stream.read()

def handler(signum, frame):
    raise IOError("The page is taking too long to read")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This read() may hang indefinitely
try:
    output = read('http://www.google.de/images/nav_logo4.png')
except IOError:
    # try to read again or print an error
    pass

signal.alarm(0)          # Disable the alarm

Answer 2

Try something like:

import socket
socket.setdefaulttimeout(5.0)
   ...
try:
   ...
except socket.timeout:
   (it timed out, retry)