very large string in memory

Question

Hi, I am writing a program for formatting 100s of MB String data (nearing a gig) into xml == And I am required to return it as a response to an HTTP (GET) request .

I am using a StringWriter/XmlWriter to build an XML of the records in a loop and returning the

using (StringWriter writer = new StringWriter())
using (writer = XmlWriter.Create(writer, settings)) //where settings are the xml props

writer.ToString() 

during testing I saw a few --out of memory exceptions-- and quite clueless on how to find a solution? do you guys have any suggestions for a memory optimized delivery of the response?

is there a memory efficient way of encoding the data? or maybe chunking the data -- I just can not think of how to return it without building the whole thing into one HUGE string object

thanks

-- a few clarifications -- this is an asp .net webservices app over a gigabit ethernet link as josh noted. I am not very familiar with it so still a bit of a learning curve.

I am using XMLWriter to create the XML and create a string out of it using String

some stats -- response xml size = about 385 megs (my data size will grow very quickly to way more than this)

string object size as calculated by a memory profiler = peaked at 605MB

and thanks to everyone who responded...

Answer 1

You'll have to return each record (or a small group of records) on their own individual GETs.

Answer 2

Use XmlTextWriter wrapped around Reponse.OutputStream to send the XML to the client and periodically flush the response. This way you never have to have more than a few mb in memory at any one time (at least for sending to the client).

Answer 3

Can't you just stream the response to the client? XmlWriter doesn't require its underlying stream to be buffered in memory. If it's ASP.NET you can use the Response.OutputStream or if it's WCF, you can use response streaming.

Answer 4

HTTP get for 1 gig? that's a lot! Perhaps you should reconsider. At least gziping the output could help.

Answer 5

You should not create XML using string manipulation.

Instead, you should use the XmlTextWriter, XmlDocument, or (in .Net 3.5) XElement classes to build an XML tree in memory, then write it directly to Response.OutputStream using an XmlTextWriter.

Writing directly to an XmlTextWriter that wraps Response.OutputStream wil be most efficient (you'll never have an entire element tree in memory at once), but will be somewhat more complicated.

By doing it this way, you will never have a single string (or array) containing the entire object, and should thus avoid OutOfMemoryExceptions.