weird problem with urllib2.proxyhandler in python 2.5

Question

in windows XP, python 2.5 and 2.6 I tested the following code:

import urllib2
proxy= urllib2.ProxyHandler({'http': '127.0.0.1:8080'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com/')

I have a proxy running at 127.0.0.1:8080 which works (I can use it in proxyswitchy and can see sites being blocked when it's not turned on).

In the above code I get a BadStatusLine exception from line 349 of httplib.py.

if I change it to a socks proxy,

proxy= urllib2.ProxyHandler({'socks': '127.0.0.1:8080'})

then it does not get used at all - in that case, the proxy is not used at all.

I got the code from the question at http://stackoverflow.com/questions/1450132/proxy-with-urllib2 and it's almost exactly the same - what could be going wrong?

Answer 1

The urllib2 ProxyHandler is not designed to support the SOCKS protocol. Perhaps this answer would help.

Answer 2

Assuming your local proxy is an HTTP proxy and not a socks proxy. Try this:

import urllib2
proxy= urllib2.ProxyHandler({'http': 'http://127.0.0.1:8080/'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com/')

Answer 3

UPDATE: I am located behind the great firewall of china. This was compounding the problem. The gfw was both breaking connections and doing DNS poisoning.

I have not managed to get any of the urllib2 solutions working. But pycurl does seem to work and it gets around the "connection reset" problem. fb/twitter were still blocked though.

Adding their IPS to my hosts file works - so for a larger scale solution, setting up a dns proxy is necessary.