Select specific child elements with BeautifulSoup

Question

I'm reading up on BeautifulSoup to screen-scrape some pretty heavy html pages. Going through the documentation of BeautifulSoup I can't seem to find a easy way to select child elements.

Given the html:

<div id="top">
  <div>Content</div>
  <div>
    <div>Content I Want</div>
  </div>
</div>

I want a easy way to to get the "Content I Want" given I have the object top. Coming to BeautifulSoup I thought it would be easy, and something like topobj.nodes[1].nodes[0].string. Instead I only see variables and functions that also return the elements together with text nodes, comments and so on.

Am I missing something? Or do I really need to resort to a long form using .find() or even worse using list comphrensions on the .contents variable.

The reason is that I don't trust the whitespace of the webpage to be the same so I want to ignore it and only traverse on elements.

Answer 1

You are more flexible with find, and to get what you want you just need to run:

node = p.find('div', text="Content I Want")

But since it might not be how you want to get there, following options might suit you better:

xml = """<div id="top"><div>Content</div><div><div>Content I Want</div></div></div>"""
from BeautifulSoup import BeautifulSoup
p = BeautifulSoup(xml)

# returns a list of texts
print p.div.div.findNextSibling().div.contents
# returns a list of texts
print p.div.div.findNextSibling().div(text=True)
# join (and strip) the values
print ''.join(s.strip() for s in p.div.div.findNextSibling().div(text=True))