Changing user agent on urllib2.urlopen

Question

How can I download a webpage with a user agent other than the default one on urllib2.urlopen?

Answer 1

Setting the User-Agent from everyone's favorite Dive Into Python.

The short story: You can use Request.add_header to do this.

You can also pass the headers as a dictionary when creating the Request itself, as the docs note:

headers should be a dictionary, and will be treated as if add_header() was called with each key and value as arguments. This is often used to “spoof” the User-Agent header, which is used by a browser to identify itself – some HTTP servers only allow requests coming from common browsers as opposed to scripts. For example, Mozilla Firefox may identify itself as "Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11", while urllib2‘s default user agent string is "Python-urllib/2.6" (on Python 2.6).

Answer 2

I answered a similar question a couple weeks ago:

http://stackoverflow.com/questions/761978/send-headers-along-in-python

There is example code in that question, but basically you can do something like this:

opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
response = opener.open('wwww.stackoverflow.com')