C# StreamWriter and StreamReader memory managment problem, why won't memory used deallocate?

Question

So I'm using a StreamReader that uses a MemoryStream to write to a StreamWriter and inside of this application but the memory usage increases by 300mb (From one of the larger inputs) and does not deallocate after Im done using it:

StreamWriter log = new StreamWriter("tempFile.txt");
log.Write(reader.ReadToEnd());
log.Close();

reader.DiscardBufferedData();
reader.Close();
reader.Dispose();
memoryStream.Dispose();
log.Dispose();
GC.Collect();

Before this and right after I get the RAM usage and before it is 300 mb less than after but I don't know why. I have done everything I can think of to release that memory considering that the only thing going on here is the data from the reader is being placed in the text file I don't see why any large amount of memory would even need to be used temporarily. Is there something I am missing?... Thanks.

Answer 1

Are you looking at the RAM used by the process itself? I wouldn't expect that to go down. The process will hang on to that memory and reuse it for further allocations. It doesn't hand it back to the operating system. That's just the way .NET tends to work.

You can potentially get it to throw away the memory with some CLR API call - but usually I wouldn't.

This doesn't mean the memory has really leaked - it's still usable by the same process. For example, if you performed the same action again it probably wouldn't need to increase the size of the heap - you'd see the memory usage stay flat at the process level. Use the CLR perfmon graphs to see the memory used within the managed heap to see if there's a genuine leak.

(There are also various different measure of how much memory an application is actually using. Which one is interesting depends on what you're trying to do.)

EDIT: If your code snippet is genuinely indicative of the code you're using, then there's a much simpler alternative: stream the data.

using (TextWriter writer = File.CreateText("tempFile.txt"))
{
    CopyText(reader, writer);
}

static void CopyText(TextReader reader, TextWriter writer)
{
    char[] buffer = new char[8192];
    int charsRead;
    while ((charsRead = reader.Read(buffer, 0, buffer.Length)) > 0)
    {
        writer.Write(buffer, 0, charsRead);
    }
}

Note that unless you're actually changing the encoding, you could do this without using a TextWriter/TextReader pair to start with.

That way you won't need to have the whole string in memory as well as its binary representation (in the MemoryStream). You'll still have the binary representation of course - do you have to write everything into the MemoryStream to start with?

Answer 2

I'm not sure if it'll help, but try some using scoping:

using (StreamReader reader = new StreamReader(somestream))
using (StreamWriter log = new StreamWriter("tempFile.txt"))
{
    log.Write(reader.ReadToEnd());
}

Answer 3

In addition to what Jon posted, the way the .NET runtime behaves acutally gives you some important benefits - keeping the memory allocated for the process is a good thing unless the system is low on memory (in which case, the runtime would probably release it).

For example, if you often need to allocate lot of memory (as in the method you posted here), then it is more efficient if the process already has the memory allocated (even though it isn't used to store any .NET objects). When you runt he method next time, the allocation will be much faster!

Anyway, a few things you could do are:

• You may try running another memory intensive application to see whether the runtime releases the memory when the system needs it
• You can use some .NET profiler to see whether there are any live objects that you would expect to be collected after running the method

Answer 4

In the segment mentioned above, all of log, reader, and memoryStream are still in scope and so can't be garbage collected. I don't know what the implmentation of Dispose on those objects is, but it's entirely likely that they're still holding much of the data in memory even after Dispose is called (since Dispose typically only closes filehandles, unmanaged memory, and such, and doesn't necessarily delete internal buffers). If you want this to work as expected, you have to let all of the referenced objects go out of scope and become unreferenced.

Really, though, you shouldn't worry about this unless memory usage is causing you explicit pain.