Refactoring - System.OutOfMemoryException

Question

I'm getting an 'System.OutOfMemoryException' when trying to generate some HTML reports.

How can I re-factor this so that the this is buffered to a file, instead of reading it all to memory, then written to a file.

There are going to be upwards for 2000+ records in the datatable, and it's already running out of memory at 2000 rows.

DetailsUpdateTemplate holds a multi-line html snippet. I'm assuming I'm creating a string that's to large.

I'm using C#, .NET 3.5

    internal static String SaveARSUpdateHTML(DataTable table, string fileName)
    {
        int recordCount = table.Rows.Count;

        Dictionary<String, object> templateCols = new Dictionary<string, object>();

        templateCols["Track"] = table.TableName;
        templateCols["ProdDate"] = DateTime.Now.ToShortDateString();
        templateCols["ProdTime"] = DateTime.Now.ToShortTimeString();
        templateCols["TotalRecords"] = recordCount;

        String detailOutput = String.Empty;
        for (int i = 0; i < table.Rows.Count; i++)
        {
            int ResultID = i + 1;
            DataRow row = table.Rows[i];
            String ReportDetails = DetailsUpdateTemplate;
            ReportDetails = ReportDetails.Replace(String.Format("{{{0}}}", "ResultID"), ResultID.ToString());
            foreach (DataColumn column in table.Columns)
            {
                String value = row[column.ColumnName].ToString();
                if (column.ColumnName.Equals("TF"))
                {
                    String display = value.Equals("no", StringComparison.CurrentCultureIgnoreCase) ? "none" : "block";
                    ReportDetails = ReportDetails.Replace(String.Format("{{{0}}}", "SuppressNewAddr"), display);
                }

                ReportDetails = ReportDetails.Replace(String.Format("{{{0}}}", column.ColumnName), value);
            } 

            detailOutput += ReportDetails;
        }

        templateCols["ReportDetails"] = detailOutput;

        String masterOut = MasterUpdateTemplate;
        foreach (KeyValuePair<string, object> pair in templateCols)
        {

            masterOut = masterOut.Replace(String.Format("{{{0}}}", pair.Key), pair.Value.ToString());
        }

        String outputFile = String.Format("{0}.htm", fileName);
        using (StreamWriter sw = new StreamWriter(outputFile))
        {
            sw.Write(masterOut);
        }

        return outputFile;
    }

Answer 1

Use StringBuilders when concatenating strings more than a handful of times.

Especially here:

detailOutput += ReportDetails;

Also use a DataReader. The DataReader will return records that support IDataRecord, which has a similar interface to DataRow.

You are hitting OOM at about the same # of rows I do when I use DataTables and DataSets in ASP.NET, where worker processes have limited memory before they will get recycles for excessive memory usage. That is why I've been aggressively switching to DataReaders where I can.

Update:

A DataReader solution would look like... (TableReaders exist, but it wouldn't buy you anything in terms of memory conservation, you'd just get an interface more similar to DataReaders)

internal static String SaveARSUpdateHTML(DbDataReader myReader, string fileName)
    {

if (myReader.HasRows)
  while (myReader.Read())
  { 
      object something = myReader["TF"];
  }

else
  Console.WriteLine("No rows returned.");

myReader.Close();

Answer 2

Use StringBuilder instead of string

Answer 3

Besides the obvious StringBuilder

Why not stream this directly to a file, just process a row at a time and then do your template replace per row. That way you do not even need to hold the entire report in memory.

The replace code on a huge template will be a memory hog.