python parsing url after string

Question

I want to extract a string from a url (link). That string is in a <h3></h3> tag.

 link = http://www.test.com/page.html

 Content of link: <h3>Text here</h3>

What would be an elegant way to first get the content/sourcecode of page.html and then exctract the link? Thanks!

Answer 1

You can use URLLib2 to retrieve the content of the URL:

http://docs.python.org/library/urllib2.html

You could then use the HTML parser in the Python libraries to find the right content:

http://docs.python.org/library/htmlparser.html

Answer 2

import urllib2
url="http://www.test.com/page.html"
page=urllib2.urlopen(url)
data=page.read()
for item in data.split("</h3>"):
    if "<h3>" in item:
         print item.split("<h3>")[1]

Answer 3

Provided the text you want is the only <h3>-wrapped text on the page, try:

from urllib2 import urlopen
from re import search
text = search(r'(?<=<h3>).+?(?=</h3>)', urlopen(link).read()).group(0)

If there are multiple <h3>-wrapped strings you can either put more details into the pattern or use re.finditer()/re.findall()

Answer 4

I'd recommend Beatiful Soup. That's a nice parser for botched HTML pages (for the most cases you don't have to worry about the page not being well-formed).