Python urllib2 timeout when using Tor as proxy?

Question

Hi,

I am using Python's urllib2 with Tor as a proxy to access a website. When I open the site's main page it works fine but when I try to view the login page (not actually log-in but just view it) I get the following error...

URLError: <urlopen error (10060, 'Operation timed out')>

To counteract this I did the following:

import socket
socket.setdefaulttimeout(None).

I still get the same timeout error.

1. Does this mean the website is timing out on the server side? (I don't know much about http processes so sorry if this is a dumb question)
2. Is there any way I can correct it so that Python is able to view the page?

Thanks, Rob

Answer 1

I don't know enough about Tor to be sure, but the timeout may not happen on the server side, but on one of the Tor nodes somewhere between you and the server. In that case there is nothing you can do other than to retry the connection.

Answer 2

According to the Python Socket Documentation the default is no timeout so specifying a value of "None" is redundant.

There are a number of possible reasons that your connection is dropping. One could be that your user-agent is "Python-urllib" which may very well be blocked. To change your user agent:

request = urllib2.Request('site.com/login')
request.add_header('User-Agent','Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.04 (jaunty) Firefox/3.5')

You may also want to try overriding the proxy settings before you try and open the url using something along the lines of:

proxy = urllib2.ProxyHandler({"http":"http://127.0.0.1:8118"})  
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

Answer 3

urllib2.urlopen(url[, data][, timeout])

The optional timeout parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS, FTP and FTPS connections.

http://docs.python.org/library/urllib2.html