how to get specific nodes in xml file with python

Question

hello
im searching for a way to get a specific tags .. from a very big xml document with python dom built in module
for example :

<AssetType longname="characters" shortname="chr" shortnames="chrs">
  <type>
    pub
  </type>
  <type>
    geo
  </type>
  <type>
    rig
  </type>
</AssetType>

<AssetType longname="camera" shortname="cam" shortnames="cams">
  <type>
    cam1
  </type>
  <type>
    cam2
  </type>
  <type>
    cam4
  </type>
</AssetType>

i want to retrieve the value of children of AssetType node who got attribute (longname= "characters" ) to have the result of 'pub','geo','rig'
please put in mind that i have more than 1000 < AssetType> nodes
thanx in advance

Answer 1

Use xml.sax module. Build your own handler and inside startElement you should check, whether name is AssetType. This way you should be able to only act, when AssetType node is processed.

Here you have example handler, which shows, how to build one (though it's not the most pretty way, at that point I didn't know all the cool tricks with Python ;-)).

Answer 2

You could use xpath, something like "//AssetType[longname='characters']/xyz".

For XPath libs in Python see http://www.somebits.com/weblog/tech/python/xpath.html

Answer 3

If you don't mind loading the whole document into memory:

from lxml import etree
data = etree.parse(fname)
result = [node.text.strip() 
    for node in data.xpath("//AssetType[@longname='characters']/type")]

You may need to remove the spaces at the beginning of your tags to make this work.

Answer 4

You could use the pulldom API to handle parsing a large file, without loading it all into memory at once. This provides a more convenient interface than using SAX with only a slight loss of performance.

It basically lets you stream the xml file until you find the bit you are interested in, then start using regular DOM operations after that.

from xml.dom import pulldom

# http://mail.python.org/pipermail/xml-sig/2005-March/011022.html
def getInnerText(oNode):
    rc = ""
    nodelist = oNode.childNodes
    for node in nodelist:
        if node.nodeType == node.TEXT_NODE:
            rc = rc + node.data
        elif node.nodeType==node.ELEMENT_NODE:
            rc = rc + getInnerText(node)   # recursive !!!
        elif node.nodeType==node.CDATA_SECTION_NODE:
            rc = rc + node.data
        else:
            # node.nodeType: PROCESSING_INSTRUCTION_NODE, COMMENT_NODE, DOCUMENT_NODE, NOTATION_NODE and so on
           pass
    return rc


# xml_file is either a filename or a file
stream = pulldom.parse(xml_file) 
for event, node in stream:
    if event == "START_ELEMENT" and node.nodeName == "AssetType":
        if node.getAttribute("longname") == "characters":
            stream.expandNode(node) # node now contains a mini-dom tree
            type_nodes = node.getElementsByTagName('type')
            for type_node in type_nodes:
                # type_text will have the value of what's inside the type text
                type_text = getInnerText(type_node)

Answer 5

Similar to eswald's solution, again stripping whitespace, again loading the document into memory, but returning the three text items at a time

from lxml import etree

data = """<AssetType longname="characters" shortname="chr" shortnames="chrs"
  <type>
    pub
  </type>
  <type>
    geo
  </type>
  <type>
    rig
  </type>
</AssetType>
"""

doc = etree.XML(data)

for asset in doc.xpath('//AssetType[@longname="characters"]'):
  threetypes = [ x.strip() for x in asset.xpath('./type/text()') ]
  print threetypes

Answer 6

Assuming your document is called assets.xml and has the following structure:

<assets>
    <AssetType>
        ...
    </AssetType>
    <AssetType>
        ...
    </AssetType>
</assets>

Then you can do the following:

from xml.etree.ElementTree import ElementTree
tree = ElementTree()
root = tree.parse("assets.xml")
for assetType in root.findall("//AssetType[@longname='characters']"):
    for type in assetType.getchildren():
        print type.text