Best approach to write huge xml data to file?

Question

Hi. I'm currently exporting a database table with huge data (100000+ records) into an xml file using XmlTextWriter class and I'm writing directly to a file on the physical drive.

_XmlTextWriterObject = new XmlTextWriter(_xmlFilePath, null);

While my code runs ok, my question is that is it the best approach? Or should I write the whole xml in memory stream first and then write the xml document in physical file from memory stream? And what are the effects on memory/ performance in both cases?

EDIT

Sorry that I could not actually convey what I meant to say.Thanks Ash for pointing out. I will indeed be using XmlTextWriter but I meant to say whether to pass a physical file path string to the XmlTextWriter constructor (or, as John suggested, to the XmlTextWriter.Create() method) or use stream based api. My current code looks like the following:

XmlWriter objXmlWriter = XmlTextWriter.Create(new BufferedStream(new FileStream(@"C:\test.xml", FileMode.Create, System.Security.AccessControl.FileSystemRights.Write, FileShare.None, 1024, FileOptions.SequentialScan)), new XmlWriterSettings { Encoding = Encoding.Unicode, Indent = true, CloseOutput = true });
using (objXmlWriter)
{
   //writing xml contents here
}

Answer 1

The rule of thumb is to use XmlWriter when the document need only be written and not worked with in memory, and to use XmlDocument (or the DOM) where you do need to work with it in memory.

Remember though, XmlWriter implements IDisposable, so do the following:

using (XmlWriter _XmlTextWriterObject = XmlWriter.Create(_xmlFilePath))
{
   // Code to do the write here
}

Answer 2

As John Saunders mentioned it is better to use XmlWriter.Create(). That is the recommendation from the MSDN. The XmlWriter.Create() method can also take an XmlWriterSettings object. There you can customize your behavior quite a bit. If you don't need validation and character checking then you can turn it off and get a bit more speed. For example

XmlWriterSettings settings = new XmlWriterSettings();
settings.CheckCharacters = false;
using (XmlWriter writer = XmlWriter.Create("path", settings)
{
    //writing code
    writer.flush();
}

Otherwise I think everything is okay.

Answer 3

While my code runs ok, my question is that is it the best approach?

As mentioned and your update, XmlWriter.Create is fine.

Or should I write the whole xml in memory stream first and then write the xml document in physical file from memory stream?

Do you have the memory to write the entire file in-memory? If you do then that approach will be faster, otherwise stream it using a FileStream which will take care of it for you.

And what are the effects on memory/ performance in both cases?

Reading the entire XML file in will use more memory, and spike the processor to start with. Streaming to disk will use more processor. But you'll need to be using a huge file for this to be noticeable given even desktop hardware now. If you're worried about the size increasing even more in the future, stick to the FileStream technique to future proof it.