Read https url from Python

Question

How do you open https url in Python?

import urllib2

url = "https://user:[email protected]/path/
f = urllib2.urlopen(url)
print f.read()

gives:

httplib.InvalidURL: nonnumeric port: '[email protected]'

Thanks in advance, Etam.

Answer 1

You cannot pass credentials to urllib2.open like that. In your case, user is interpreted as the domain name, while [email protected] is interpreted as the port number.

Answer 2

Please read about the urllib2 password manager and the basic authentication handler as well as the digest authentication handler.

http://docs.python.org/library/urllib2.html#abstractbasicauthhandler-objects

http://docs.python.org/library/urllib2.html#httpdigestauthhandler-objects

Your urllib2 script must actually provide enough information to do HTTP authentication. Usernames, Passwords, Domains, etc.

Answer 3

If you want to pass username and password information to urllib2 you'll need to use an HTTPBasicAuthHandler.

Here's a tutorial showing you how to do it.

Answer 4

This has never failed me

import urllib2, base64
username = 'foo'
password = 'bar'
auth_encoded = base64.encodestring('%s:%s' % (username, password))[:-1]

req = urllib2.Request('https://somewebsite.com')
req.add_header('Authorization', 'Basic %s' % auth_encoded)
try:
    response = urllib2.urlopen(req)
except urllib2.HTTPError, http_e:
    # etc...
    pass