Python HTML scraping

Question

Hey,

It's not really scraping, I'm just trying to find the URLs in a web page where the class has a specific value. For example:

<a class="myClass" href="/url/7df028f508c4685ddf65987a0bd6f22e">

I want to get the href value. Any ideas on how to do this? Maybe regex? Could you post some example code? I'm guessing html scraping libs, such as BeautifulSoup, are a bit of overkill just for this...

Huge thanks!

Answer 1

Aargh, not regex for parsing HTML!

Luckily in Python we have BeautifulSoup or lxml to do that job for us.

Answer 2

Regex is usally a bad idea, try using BeautifulSoup

Quick example:

html = #get html
soup = BeautifulSoup(html)
links = soup.findAll('a', attrs={'class': 'myclass'})
for link in links:
    #process link

Answer 3

Regex would be a bad choice. HTML is not a regular language. How about Beautiful Soup?

Answer 4

read Parsing Html The Cthulhu Way http://www.codinghorror.com/blog/archives/001311.html

Answer 5

Regex should not be used to parse HTML. See the first answer to this question for an explanation :)

+1 for BeautifulSoup.

Answer 6

The thing is I know the structure of the HTML page, and I just want to find that specific kind of links (where class="myclass"). BeautifulSoup anyway?

Answer 7

If your task is just this simple, just use string manipulation (without even regex)

f=open("htmlfile")
for line in f:
    if "<a class" in line and "myClass" in line and "href" in line:
        s = line [ line.index("href") + len('href="') : ]
        print s[:s.index('">')]
f.close()

HTML parsers is not a must for such cases.