Query xml file without loading it into memory?

Question

I am looking for something other than XmlReader.

I want to apply a query and load only the data that is needed.

Actually I want to load as little of the xml as possible.

Maybe some kind of XQuery utility/class could do the trick.

Answer 1

You can mix XmlReader with LINQ to XML to get the best of both worlds. The trick is to use LINQ to XML to load just inner nodes into memory.

For instance, suppose you have a XML file with the following structure:

<log>
  <logentry id="1">
    <date>...</date>
    <source>...</source>
    ...
  </logentry>
  ...
</log>

Imagine there are a million logentry elements, and you want to find the elements where the source element contains a particular string. First, write the following method:

Enumerable<LogEntry> ReadLogEntries (XmlReader r)
{
  r.ReadStartElement ("log");
  while (r.Name == "logentry")
  {
    XElement logEntry = (XElement) XNode.ReadFrom (r);
    yield return new LogEntry 
    {
      ID = (int) logEntry.Attribute ("id"),
      Date = (DateTime) logEntry.Element ("date"),
      Source = (string) logEntry.Element ("source")
    }
  }
  r.ReadEndElement();
}

class LogEntry
{
  public int ID;
  public DateTime Date;
  public string Source;
}

Then you can query the XML file (without loading it all into memory) as follows:

from l in ReadLogEntries (reader)
where l.Source.Contains ("foo")
select new { l.ID, l.Date }