Tips on Dealing with Large Strings With Regards to Memory Usage

Question

Hi

How can I force a shrink of a DataTable and/or List so that I can free memory efficiently? I am currently removing the processed row from the DataSet every iteration of the loop, but I'm not sure if the memory is being released.

for (int i = m_TotalNumberOfLocalRows - 1; i >= 0; i--)
{
   dr = dt.Rows[i];
   // Do stuff
   dt.Rows.Remove(dr);
}

If this doesn't shrink the DataTable's footprint in memory I can use a List. I'm only using the DataTable as a store for the DataRows, I could use whatever collection or storage mechanism that would be low on memory and be able to release memory every x iterations.

Thank you.

---

---

Edit: After doing some memory profiling after reading http://stackoverflow.com/questions/3927/what-are-some-good-net-profilers I've discovered that the main consumer of memory are strings.

We are doing lots of user output during this process, and the memory consumption is cycling between approximately 170MB-230MB, peaking at around 300MB. I'm using a StringBuilder with an initial size of 20971520 to hold the output/log of what is happening and after one percent of the total number of records have been processed I'm setting a DevExpress MemoEdit control's Text property to the StringBuilder.ToString(). I've found that this method is quicker than appending the StringBuilder.ToString() to the MemoEdit.Text (obviously the logic w.r.t. the StringBuilder is different between appending and setting the MemoEdit.Text)

I've also found that instead of recreating the StringBuilder(20971520) it's easier on memory and quicker to execute to just StringBuilder.Remove(0, StringBuilder.Length)

Are there any tips that you could share to improve performance when working with large strings (the log file that it written out that contains the log is approximately 12.2MB for about 30 000 records)?

Note: I have changed the title of the question and the tags.
Old title:How can I Force a Shrink of a DataTable and/or a List to Release Memory?
Old tags: list datatable c# .net memory

Answer 1

Try forcing a garbage collector pass:

GC.Collect();

If you want to make sure, that all objects are finalized before your code execution continues, call

GC.WaitForPendingFinalizers();

right after GC.Collect()

---

EDIT: As people mentioned in the comments below, it is widely considered a bad practice to call the Garbage Collector directly. Nevertheless, this sould achieve the goal of freeing the unnused memory of your deleted rows.

Answer 2

Stick to the DataTable and remove unnecessary rows as in your example.

By doing this you can't control memory usage: this is done by the CLR Garbage Collector.

Answer 3

Do you have an explicit need to manage this directly? The garbage collector manages this for you.

Answer 4

Unless you are having a problem with memory, don't try and manually free it by calling the garbage collector. The run time will handle it for you and 99% of the time be more efficient at it than you will by trying to guess when the optimal time is.

What you have to remember is that when you call GC.Collect(), it runs against all the levels of the Garbage Collection and "tidies up" all objects that need to need to be freed. You will most likely be spending processor time etc. handling something that doesn't need to be done at that point in time.

If you absolutely have to the command is GC.Collect()

http://msdn.microsoft.com/en-us/library/xe0c2357.aspx

http://www.developer.com/net/csharp/article.php/3343191/C-Tip-Forcing-Garbage-Collection-in-NET.htm