I am using a proxy and following is the code.
20 req = urllib2.Request(url)
21 # run the request for each proxy
22 # now set the proxy
23 req.set_proxy(proxy, "http")
24 req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3')
25 req.add_header('Referer', 'http://www.google.com/')
26 # time it
27 start = time.time()
28 # try to open the URL
29
30 time.sleep(maxwait*random.random())
31 #response = urllib2.urlopen(req)
32 opener = urllib2.build_opener()
33 response = opener.open(req).read()
Works fine on one machine (with eclipse and Firefox directly) but fails while running it at the terminal on another machine (different IP) with the following error.
urllib2.HTTPError: HTTP Error 403: Forbidden
Any leads? Thanks guys!