Get webpage contents with Python?

Question

I'm using Python 3.1, if that helps.

Anyways, I'm trying to get the contents of this webpage. I Googled for a little bit and tried different things, but they didn't work. I'm guessing that this should be an easy task, but...I can't get it. :/.

import urllib.request
page = urllib.request.urlopen('http://hiscore.runescape.com/index_lite.ws?player=zezima')
print(page.read())

Thank you, Jason. :D.

Answer 1

You can use urlib2 and parse the HTML yourself.

Or try Beautiful Soup to do some of the parsing for you.

Answer 2

Python 3 is the future, but if you are trying to do something right now, I would suggest sticking with 2.x - It has tons more documentation and examples you can find on the web.

Was there any reason for using python 3?

Answer 3

Because you're using Python 3.1, you need to use the new Python 3.1 APIs

Try

urllib.request.urlopen('http://www.python.org/')

Alternately, it looks like you're working from Python 2 examples. Write it in Python 2, then use the 2to3 tool to convert it. On Windows, 2to3.py is in \python31\tools\scripts. Can someone else point out where to find 2to3.py on other platforms?

Answer 4

Mechanize is a great package for "acting like a browser", if you want to handle cookie state, etc.

http://wwwsearch.sourceforge.net/mechanize/