How can I search a large XML data set?

Question

I have a DataModule with XML and I need do a search...

Unfortunately there are more than 300,000 records and I can't make a loop to check one-by-one.

Is it possible to make a query without using a database?

Is there another solution?

Answer 1

XML's fine for small amounts of information, but for a dataset that big, a relational database is really the only sane choice, especially if you need to be able to query it.

Answer 2

You can search using something like XPath: however, that would just mean that the XPath implementation does the searching on your behalf (which doesn't necessarily improve performance).

Answer 3

There are a number of in-memory databases that may be useful. At least you could then index and query the data as required. One I know of is from components4developers.com. David

Answer 4

I think it is probably important to ask Why are you using XML to store 300k records?. As XML is not the most efficient format to manipulate data with.

If you're stuck with XML then you might be best to read the XML file into some sort of database (you might get away with an in memory table, but then again you might run out of memory). I think if you use a TXMLDocument object to load the XML file into you'll either have a serious performance issue or run out of memory (I had trouble when I was playing with a 250k record xml file awhile back).

You might be able to use the MSXML DOM directly (you can probably import the type library) or use SAX which will allow you to parse it sequentially, neither of which I have had much experience with.

Answer 5

You don't say how you are implementing the datasource. I have used TClientDataSet connected via TXMLTransformProvider (OK not for 300K records) but for a few thousand. and simply setting the filter and filtered properties seems to "Query" it just fine...

Or have I missed something?