How do I ask for an authenticated url directly with python

Question

I want to get to an authenticated page using urllib2. I'm hoping there's a hack to do it directly. something like:

urllib2.urlopen('http://username:pwd@server/page')

If not, how do I use authentication?

Answer 1

It depends on the type of authentication used.

• A simple example is Http Authentication
• If the site uses cookies for auth you need to add a cookiejar and login over http
• there are many more auth schemes, so find out which you need.

Answer 2

AFAIK, there isn't a trivial way of doing this. Basically, you make a request and the server responds with a 401 authorization required, which urllib2 translates into an exception.

  Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python25\lib\urllib2.py", line 124, in urlopen
    return _opener.open(url, data)
  File "C:\Python25\lib\urllib2.py", line 387, in open
    response = meth(req, response)
  File "C:\Python25\lib\urllib2.py", line 498, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python25\lib\urllib2.py", line 425, in error
    return self._call_chain(*args)
  File "C:\Python25\lib\urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "C:\Python25\lib\urllib2.py", line 506, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
  urllib2.HTTPError: HTTP Error 401: Authorization Required

You will have to catch this exception, create a urllib2.HTTPPasswordManager object, add the username and password to the HTTPPasswordManager, create a urllib2.HTTPBasicAuthHandler object, create an opener object and finally fetch the url using the opener. Code and a tutorial is available here: http://www.voidspace.org.uk/python/articles/urllib2.shtml#id5