Beautiful Soup: Get the Contents of Sub-Nodes

Question

Hello,

I have following python code:

def scrapeSite(urlToCheck):
    html = urllib2.urlopen(urlToCheck).read()
    from BeautifulSoup import BeautifulSoup
    soup = BeautifulSoup(html)
    tdtags = soup.findAll('td', { "class" : "c" })
    for t in tdtags:
            print t.encode('latin1')

This will return me following html code:

<td class="c">
<a href="more.asp">FOO</a>
</td>
<td class="c">
<a href="alotmore.asp">BAR</a>
</td>

I'd like to get the text between the a-Node (e.g. FOO or BAR), which would be t.contents.contents. Unfortunately it doesn't work that easy :) Does anyone have an idea how to solve that?

Thanks a lot, any help is appreciated!

Cheers, Joseph

Answer 1

In this case, you can use t.contents[1].contents[0] to get FOO and BAR.

The thing is that contents returns a list with all elements (Tags and NavigableStrings), if you print contents, you can see it's something like

[u'\n', <a href="more.asp">FOO</a>, u'\n']

So, to get to the actual tag you need to access contents[1] (if you have the exact same contents, this can vary depending on the source HTML), after you've find the proper index you can use contents[0] afterwards to get the string inside the a tag.

Now, as this depends on the exact contents of the HTML source, it's very fragile. A more generic and robust solution would be to use find() again to find the 'a' tag, via t.find('a') and then use the contents list to get the values in it t.find('a').contents[0] or just t.find('a').contents to get the whole list.

Answer 2

For your specific example, pyparsing's makeHTMLTags can be useful, since they are tolerant of many HTML variabilities in HTML tags, but provide a handy structure to the results:

html = """
<td class="c"> 
<a href="more.asp">FOO</a> 
</td> 
<td class="c"> 
<a href="alotmore.asp">BAR</a> 
</td> 
<td class="d"> 
<a href="alotmore.asp">BAZZ</a> 
</td> 
"""

from pyparsing import *

td,tdEnd = makeHTMLTags("td")
a,aEnd = makeHTMLTags("a")
td.setParseAction(withAttribute(**{"class":"c"}))

pattern = td + a("anchor") + SkipTo(aEnd)("aBody") + aEnd + tdEnd

for t,_,_ in pattern.scanString(html):
    print t.aBody, '->', t.anchor.href

prints:

FOO -> more.asp
BAR -> alotmore.asp