OutOfMemoryException when creating huge string in ASP.NET

Question

When exporting a lot of data to a string (csv format), I get a OutOfMemoryException. What's the best way to tackle this? The string is returned to a Flex Application.

What I'd do is export the csv to the server disk and give back an url to Flex. Like this, I can flush the stream writing to the disk.

Update:

String is build with a StringBuilder:

StringBuilder stringbuilder = new StringBuilder();
string delimiter = ";";
bool showUserData = true;

// Get the data from the sessionwarehouse
List<DwhSessionDto> collection 
     =_dwhSessionRepository.GetByTreeStructureId(treeStructureId);

// ADD THE HEADERS
stringbuilder.Append("UserId" + delimiter);
if (showUserData)
{
    stringbuilder.Append("FirstName" + delimiter);
    stringbuilder.Append("LastName" + delimiter);
}
stringbuilder.Append("SessionId" + delimiter);
stringbuilder.Append("TreeStructureId" + delimiter);
stringbuilder.Append("Name" + delimiter);
stringbuilder.Append("Score" + delimiter);
stringbuilder.Append("MaximumScore" + delimiter);
stringbuilder.Append("MinimumScore" + delimiter);
stringbuilder.Append("ReducedScore" + delimiter);
stringbuilder.Append("ReducedMaximumScore" + delimiter);
stringbuilder.Append("Duration" + delimiter);
stringbuilder.AppendLine("Category" + delimiter);

foreach (var dwhSessionDto in collection)
{
    stringbuilder.Append(
        getPackageItemsInCsvFromDwhSessionDto(
            dwhSessionDto, delimiter, showUserData));
}

return stringbuilder.ToString();

The string is sent back to Flex like this:

var contentType = "text/csv";
string result = exportSessionService.ExportPackage(treeStructureId);
// Write the export to the response
_context.Response.ContentType = contentType;
_context.Response.AddHeader("content-disposition", 
    String.Format("attachment; filename={0}", treeStructureId + ".csv"));

// do not Omit the Vary star so the caching at the client side will be disabled
_context.Response.Cache.SetOmitVaryStar(false);
_context.Response.Cache.SetCacheability(HttpCacheability.NoCache);

if (!String.IsNullOrEmpty(result))
{
    _context.Response.Output.Write(result);
    _context.Response.Output.Close();
}
else
{
    _context.Response.Output.Write("No logs");
}

Answer 1

For me its better to create a generic handler page (ashx), and send all your data directly to flex.

Direct to the stream...

Here are a nice article about this issue

Answer 2

First, how much would more RAM cost? These days hardware is cheaper than programmers.

Without a bit more to go on, more concrete recommendations are a bit of a trick. But you want to avoid actually building the CSV as a string but rather stream it from the data source to disk and not hold the entire thing in memory.

Answer 3

What I'd do is export the csv to the server disk

Sounds like the right idea. But are you creating the complete string first? If you write line by line you shouldn't get OOM

Edit, after seeing the code

You are only using StringBuilder.Append(x), so you can restructure your code a little to replace that with _context.Response.Output.Write(x). That might sacrifice a little bit of your separation but it's for a good cause.

And if you want to keep using a StringBuilder, it would help a lot if you could estimate the resulting size. Add a generous margin and use

   StringBuilder stringbuilder = new StringBuilder(estimate);

This saves on growing (copying) of the StringBuilder and reduces both memory use and fragmentation on the LOH.

Answer 4

Your CSV strings are growing to over 80000 bytes and ending up on the LargeObjectHeap. The LOH is not garbage collected in the same way as other generations and can fragment over time, such as after many requests to your server or if you use string concatenation (naughty!) to build this csv data. The result is that your program reserves much more memory than it's actually using and an OutOfMemory exception is thrown.

The fix in this instance is to write your csv data directly to the Response stream rather than string variables or even StringBuilder. Not only will this avoid the Large Object Heap, but it will keep your overall memory use lower and start pushing data out to your user faster.

Answer 5

String is created with a StringBuilder and communicated back to Flex Application with HttpContext.Reponse.Output.Write(result); – Lieven Cardoen

So what you're basically doing is this:

StringBuilder str = new StringBuilder();
while(whatever)
{
    str.Append(thisNextLineIGuess);
}
Response.Output.Write(str.ToString());

Correct? That may not be your exact code, but it sounds like a good approximation.

Then the solution is simple:

while(whatever)
{
    Response.Output.Write(thisNextLineIGuess);
}

Streaming is usually better than buffering the whole thing and sending it out as a lump.