What is the fastest way to remove nodes from a large XML file using .net

Question

I am working working with very large XML files (100s of MBs). The tree is fairly simple

<items>
  <item>
    <column1>ABC</column1>
    <column2>DEF</column2>
  </item>
  <item>
    <column1>GHI</column1>
    <column2>KLM</column2>
  </item>
</items>

I need to parse this document, and remove some <item> elements. So far, the best peerformance I achieved is using XmlReader, caching each <item> in memory and the writing it back using XmlWriter out if it meets the criteria, and simply ignoring it if it doesn't. Is there anyting i can do to make it faster?

Answer 1

You could use perl or shell scripting to replace out the required items if you can write a quick regular expression to get rid of it. That would avoid loading the whole thing into memory and writing it back out.

Answer 2

You might be able to save a step by implementing a subclass of XmlReader whose Read method skips over the item elements you're not interested in. Right now, you seem to have two steps: reading and filtering the document with an XmlReader and then using XmlWriter to write it to something that you presumably then read it from. Subclassing XmlReader eliminates that second step; you use the subclassed XmlReader as the input to your XSLT transform or XmlDocument or whatever, and it never builds an intermediate representation of the filtered XML document.

Answer 3

see if you can use xpath querys to determine what you want to and dont want to read with that xmldocument object....look into the following methods of that class SelectSingleNode() which returns an XmlNode object... SelectNodes() which returns an XmlNodeList object.... see if that helps....

Answer 4

This URL has the answer you look for

http://stackoverflow.com/questions/62423/how-to-update-large-xml-file