FileStream and memory usage

Question

I've written the following program which purpose is to create a file of a give size with some random data in it. The program works fine and does what it's suppose to do. However, I don't understand why it consumes 5GB of RAM (see screenshot of my Task Manager). While I am writing the file with random data, I am not creating new objects. What am I missing? I would expect this program to take no memory at all.

The big problem I have right now is that in the middle on the file generation, the machine is dying...

class Program
{
    static void Main(string[] args)
    {
        CreateFile("test.dat", 10 * 1024 * 1024);
    }

    public static void CreateFile(string path, long approximativeFileSizeInKb)
    {
        RandomNumberGenerator randomNumber = RandomNumberGenerator.Create();

        byte[] randomData = new byte[64 * 1024];

        int numberOfIteration = 0;
        randomNumber.GetNonZeroBytes(randomData);

        using (FileStream fs = File.Create(path, 64 * 1024))
        {
            while (numberOfIteration++ * 64 < approximativeFileSizeInKb)
            {
                fs.Write(randomData, 0, randomData.Length);
            }
        }
    }
}

Answer 1

The FileStream maintains a memory buffer, allowing your program to output as fast as it can. Your program can stuff data into the buffer MUCH faster than that buffer can be written to disk, which is where the memory jump comes in.

The memory actually used does seem more than a little off, though; you're generating a 10MB file (in 64KB chunks) and using about 5GB of memory to do so. Is there anything else to this program than what's in the code snippet? Are you running it multiple times?

Answer 2

Windows seems to be using file system caching... it's not the app

Answer 3

You have a lot of RAM, more than you need. Until you gave it something useful to do: storing that large file in the file system cache. The benefit is that your program runs very fast, even though it is writing an enormous file. That works up to a point until the cache fills up because the disk cannot keep up. Then the perf of your program falls off a cliff, only being able to write as fast as the disk lets it.

Answer 4

Like Gonzalo, I ran your code on my system and only saw a 1GB increase in memory usage.

Do you have anti-virus turned on? The AV may be scanning the .dat file as it's being written, causing the data to be buffered in memory while the scan takes place, which causes the huge increase in memory usage. If you suspect AV to be part of the problem, try changing the file extension to something other than .dat (e.g. .txt).

Another thing to try is adding a call to fs.Flush() after fs.Write(...).

Answer 5

Lucero is right.

I executed the code and definitely the console application is not using the memory, it keeps at 23 Mb. The memory is used by Windows Cache. To confirm it, I debugged and paused the process at some point. What I saw was how memory usage slowly goes down. Note that a paused process can't free any memory at all.

Answer 6

Change your line which reads:

using (FileStream fs = File.Create(path, 64 * 1024))

to

using (FileStream fs = File.Create(path, 64 * 1024, FileOptions.WriteThrough))

and see how that does for you.

Answer 7

File system writes are always buffered by the OS.

You are calling FileSystem.Write faster than your hardware can handle the writes, thus the OS is caching all of your writes.

Even if you called FileSystem.Flush, you would still be writing faster than your hardware can handle the writes.

Get a faster hard disk subsystem. Preferrable an RAID controller with lots of on board memory connected to a large RAID 5 or 6 array with server based hard drives with 64MB caches set with write buffering.

(To alleviate this behavior add the flag FileOptions.WriteThrough to your File.Create call.)