Getting non-contiguous text with lxml / ElementTree

Question

Suppose I have this sort of HTML from which I need to select "text2" using lxml / ElementTree:

<div>text1<span>childtext1</span>text2<span>childtext2</span>text3</div>

If I already have the div element as mydiv, then mydiv.text returns just "text1".

Using itertext() seems problematic or cumbersome at best since it walks the entire tree under the div.

Is there any simple/elegant way to extract a non-first text chunk from an element?

Answer 1

Such text will be in the tail attributes of the children of your element. If your element were in elem then:

elem[0].tail

Would give you the tail text of the first child within the element, in your case the "text2" you are looking for.

Answer 2

As llasram said, any text not in the text attribute will be in the tail attributes of the child nodes.

As an example, here's the simplest way to extract all of the text chunks (first and otherwise) in a node:

html = '<div>text1<span>childtext1</span>text2<span>childtext2</span>text3</div>'

import lxml.html    # ...or lxml.etree as appropriate
div = lxml.html.fromstring(html)

texts = [div.text] + [child.tail for child in div]
# Result: texts == ['text1', 'text2', 'text3']
# ...and you are guaranteed that div[x].tail == texts[x+1]
# (which can be useful if you need to access or modify the DOM)

If you'd rather sacrifice that relation in order to prevent texts from potentially containing empty strings, you could use this instead:

texts = [div.text] + [child.tail for child in div if child.tail]

I haven't tested this with plain old stdlib ElementTree, but it should work with that too. (Something that only occurred to me once I saw Shane Holloway's lxml-specific solution) I just prefer LXML because it's got better support for HTML's ideosyncracies and I usually already have it installed for lxml.html.clean

Answer 3

Well, lxml.etree provides full XPath support, which allows you to address the text items:

>>> import lxml.etree
>>> fragment = '<div>text1<span>childtext1</span>text2<span>childtext2</span>text3</div>'
>>> div = lxml.etree.fromstring(fragment)
>>> div.xpath('./text()')
['text1', 'text2', 'text3']