Send headers along in python

Question

I have the following python script and I would like to send "fake" header information along so that my application acts as if it is firefox. How could I do that?

import urllib, urllib2, cookielib

username = '****'

password = '****' 

login_user = urllib.urlencode({'password' : password, 'username' : username})

jar = cookielib.FileCookieJar("cookies")

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar))

response = opener.open("http://www.***.com")

response = opener.open("http://www.***.com/login.php")

response = opener.open("http://www.***.com/welcome.php", login_user)

Answer 1

You have to get a bit more low-level to be able to do that.

request = urllib2.Request('http://stackoverflow.com')
request.add_header('User-Agent', 'FIREFOX LOL')
opener = urllib2.build_opener()
data = opener.open(request).read()
print data

Not tested.

Answer 2

Use the addheaders() function on your opener object.
Just add this one line after you create your opener, before you start opening pages:

opener.addheaders = [('User-agent', 'Mozilla/5.0')]

http://docs.python.org/library/urllib2.html (it's at the bottom of this document)

Answer 3

FWIW, depending on just how precisely you want to mimic Firefox, setting the User-Agent may not be enough (though that is probably sufficient for most cases). To make your script to look like 'normal' web browsing, you might want to set an appropriate Referer and make additional requests for the rest of the page content (Javascript/CSS/Images/Flash/etc). Something to thing about, though perhaps not appropriate to your particular situation.