Python's urllib2 don't work on some sites

Question

I found that you can't read from some sites using Python's urllib2(or urllib). An example...

urllib2.urlopen("http://www.dafont.com/").read()
# Returns ''

These sites works when you visit the site. I can even scrap them using PHP(didn't try other languages). I have seen other sites with the same issue - but can't remember the URL at the moment.

My questions are...

1. What is the cause of this issue?
2. Any workaround for this issue?

Answer 1

I'm the guy who posted the question. I have some suspicions - but not sure about them - that's why I posted the question here.

What is the cause of this issue?

I think its due to the host blocking the urllib library using robot.txt or htaccess. But not sure about it. Not even sure if its possible.

Any workaround for this issue?

If you are in Unix, this will work...

contents = commands.getoutput("curl -s '"+url+"'")

Answer 2

I believe it gets blocked by the User-Agent. You can change User-Agent using the following sample code:

USERAGENT = 'something'
HEADERS = {'User-Agent': USERAGENT}

req = urllib2.Request(URL_HERE, headers=HEADERS)
f = urllib2.urlopen(req)
s = f.read()
f.close()

Answer 3

Try setting a different user agent. http://stackoverflow.com/questions/802134/changing-user-agent-on-urllib2-urlopen