Python proxy checker, change to threaded version

Question

Hello ALL,

i have some python proxy checker.

and to speed up check, i was decided change to multithreaded version,

and thread module is first for me, i was tried several times to convert to thread version

and look for many info, but it not so much easy for novice python programmer.

if anyone can help me really much appreciate!!

thanks in advance!

import urllib2, socket

socket.setdefaulttimeout(180)
# read the list of proxy IPs in proxyList
proxyList = open('listproxy.txt').read()

def is_bad_proxy(pip):    
    try:        
        proxy_handler = urllib2.ProxyHandler({'http': pip})        
        opener = urllib2.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib2.install_opener(opener)        
        req=urllib2.Request('http://www.yahoo.com')  # <---check whether proxy alive 
        sock=urllib2.urlopen(req)
    except urllib2.HTTPError, e:        
        print 'Error code: ', e.code
        return e.code
    except Exception, detail:

        print "ERROR:", detail
        return 1
    return 0


for item in proxyList:
    if is_bad_proxy(item):
        print "Bad Proxy", item
    else:
        print item, "is working"

Answer 1

urllib2.install_opener() function installs global opener, i.e. it's not thread safe. So don't use it and call opener.open() method instead of global urllib2.urlopen() function. Also use Queue class from Queue module to hold the list of proxies to check. The rest of your code is OK to use in threaded mode.