setting refferal url in python urllib.urlretrieve.

Question

I am using urllib.urlretrieve in python to download websites. Though some websites seem to not want me to download them, unless they have a proper referrer from their own site. Does anybody know of a way I can set a referrer in one of python's libraries or a external one to.

Answer 1

if you see the urllib2 docs here. there is example of how to put referrer to the header.

Answer 2

urllib makes it hard to send arbitrary headers with the request; you could use urllib2, which lets you build and send a Request object with arbitrary headers (including of course the -- alas sadly spelled;-) -- Referer). Doesn't offer urlretrieve, but it's easy to just urlopen as you with and copy the resulting file-like object to disk if you want (directly, or e.g. via shutil functions).

Answer 3

import urllib2
req = urllib2.Request('http://www.example.com/')
req.add_header('Referer', 'http://www.python.org/')
r = urllib2.urlopen(req)

adopted from http://docs.python.org/library/urllib2.html