Easy way of Multiple XML file search

Question

Hi

I want to search for an element value in all the XML files(assume 200+) in a folder using C#.

My scenario is each file will contain multiple item tags.So i have to check all item tags for User Selected SearchValue. Eg: ABC123

Currently i am using foreach loop and it's taking longtime.

Could you please suggest me a better option to get result much faster

Following is my current code implementation.

string[] arrFiles = Directory.GetFiles(temFolder, "*.xml");
            foreach (string file in arrFiles)
            {
                XmlDocument doc = new XmlDocument();
                doc.Load(file);
                XmlNodeList lstEquip = doc.SelectNodes("scene/PackedUnit/Items/ItemCode");
                foreach (XmlNode xnEquip in lstEquip)
                {
                    if (xnEquip.InnerText.ToUpper() == equipCode.ToUpper())
                    {
                        String[] strings = file.Split('\\');
                        string fileName = strings[strings.Count() - 1];
                        fileName = fileName.Replace(".xml", "");
                        lstSubContainers.Add(fileName);
                        break;
                    }
                }
            }

Thanks In Advance.

Answer 1

Well, the first thing to work out is why it's taking a long time. You haven't provided any code, so it's hard to say what's going on.

One option is to parallelize the operation, using a pool of tasks each working on a single document at a time. In an ideal world you'd probably read from the files on a single thread (to prevent thrashing) and supply the files to the pool as you read them - but just reading in multiple threads it probably a good starting point. Using .NET 4's Parallel Extensions libraries would make this reasonably straightforward.

Personally I like the LINQ to XML API for querying, rather than using the "old" XmlElement etc API, but it's up to you. I wouldn't expect it to make much difference. Using XmlReader instead could be faster, avoiding creating as much garbage - but I would try to find out where the time is going in the "simple" code first. (I personally find XmlReader rather harder to use correctly than the "whole document in memory" APIs.)

Answer 2

If you're doing forward-only reading and not manipulating the Xml in anyway, switching to an XmlReader should speed up the processing, although I can't imagine it will really make a massive difference (maybe a second or two atmost) with the file sizes you have.

I've recently had to parse a 250mb XML file using LINQ-to-XML in Silverlight (a test app) and that took seconds to do. What is your machine?