"HTTP Error 409: Conflict" when using urllib.request.urlopen()

Question

Under Python 3.1, when trying to run this code:

from urllib import request

def test():
    request.urlopen("http://www.google.com")

test()

I get an HTTP 409 error. The stack trace is:

Traceback (most recent call last):
  File "C:\Users\Beau\Python\pokescrape.py", line 6, in <module>
    test()
  File "C:\Users\Beau\Python\pokescrape.py", line 4, in test
    request.urlopen("http://www.google.com")
  File "C:\Program Files\Python\lib\urllib\request.py", line 119, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Program Files\Python\lib\urllib\request.py", line 353, in open
    response = meth(req, response)
  File "C:\Program Files\Python\lib\urllib\request.py", line 465, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files\Python\lib\urllib\request.py", line 385, in error
    result = self._call_chain(*args)
  File "C:\Program Files\Python\lib\urllib\request.py", line 325, in _call_chain
    result = func(*args)
  File "C:\Program Files\Python\lib\urllib\request.py", line 560, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Program Files\Python\lib\urllib\request.py", line 353, in open
    response = meth(req, response)
  File "C:\Program Files\Python\lib\urllib\request.py", line 465, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files\Python\lib\urllib\request.py", line 391, in error
    return self._call_chain(*args)
  File "C:\Program Files\Python\lib\urllib\request.py", line 325, in _call_chain
    result = func(*args)
  File "C:\Program Files\Python\lib\urllib\request.py", line 473, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

And the actual error, as printed, is:

urllib.error.HTTPError: HTTP Error 409: Conflict

I'm behind a proxy set via a configuration script; I've had no issues regarding internet connection.

Why on earth am I getting an HTTP 409 error?

Answer 1

The HTTP error you're seeing is one the remote end (or the proxy) is giving you. HTTP Error 409 is indeed 'Conflict', which usually means there were conflicting requests being made. If you are indeed using a proxy I would suspect that being the source of the 409, but more debugging would be in order. Either use a tool like wireshark to analyze the actual traffic, or use http.client.HTTPConnection directly and turn on its debugging.

Answer 2

It turns out I had to manually set the proxy in-code. I'm assuming this was because I was using an automated proxy script.

For anybody with a similar issue, here's the code I used:

from urllib import request
import random

PROXY_URL = "http://wwwcache-{}.lancs.ac.uk:8080/"

def setLancsProxy():
    httpProxy = PROXY_URL.format(random.randrange(4))
    proxy = request.ProxyHandler({"http" : httpProxy})
    opener = request.build_opener(proxy)
    request.install_opener(opener)

I found http://groups.google.com/group/comp.lang.python/browse_thread/thread/a9db4a2f398ee3a4 and http://www.wkoorts.com/wkblog/2008/10/27/python-proxy-client-connections-requiring-authentication-using-urllib2-proxyhandler/ most helpful in dealing with the obscure issue.

Answer 3

I was running into this problem too (also from Lancaster, as it happens) and found that if I set the environment variable http_proxy, Python would use it. In this case (on Windows) it would be:

set http_proxy=http://wwwcache.lancs.ac.uk:8080

and I believe the same would work on *nix, but I haven't tried it:

export $http_proxy=http://wwwcache.lancs.ac.uk:8080