Data Import/Export of large data sets with .net

Question

Hi,

has anyone experience with import and export of large collections (from a database in this case) with .net.

Using the database export itself is not possible because there are multiple database backends supported and i need this for a kind of platform independent export/import.

The problem here is that the XmlSerializer or DataContractSerializer classes read the data all at once for deserialization - but since the data sets can get extremely large this is not feasible. Are there any solutions which build upon the existing serialization infrastructure but support iterative reading of the files?

Thanks

Answer 1

Take a look at FileHelpers. I've used that library in the past and it read and validated (via attributes attached to my import class members) about 25k records in a couple of seconds.