Windows system cache and huge files.

Question

Hi, i have a problem with windows system cache. Sample code:

int main(int argc, char* argv[])
{
    HANDLE file_ = INVALID_HANDLE_VALUE;
    file_ = CreateFile(
                            "test_file.txt",
                            GENERIC_WRITE, 
                            FILE_SHARE_READ, 
                            0, 
                            OPEN_ALWAYS, 
                            FILE_FLAG_SEQUENTIAL_SCAN, 
                            NULL
                            );
    if (file_ == INVALID_HANDLE_VALUE || file_ == NULL)
    {
        std::cout << "CreateFile error " << GetLastError() << std::endl;
        return GetLastError();
    }
    int counter = 0;
    DWORD io_bytes = 0;
    while(true)
    {
        char buffer[0x1000];
        int len = _snprintf_s(buffer, 0x1000, 0xffff, "test message %d\r\n", counter);
        counter++;
        if ( !WriteFile(file_, buffer, len, &io_bytes, NULL) )
        {
            std::cout << "WriteFile error " << GetLastError() << std::endl;
            return GetLastError();
        }
        if (counter > 10000000)
        {
            system("pause");
            return 0;
        }
    }
}

if you run this code and look at system cache size, all will be ok. But if you open this file(test_file.txt) for reading in some viewer(for exemple with lister plugin for total commander) while this program is running, you will see, that system cache size is growing, even if you already close your viewer program. This looks like memory leak. Is this behavior normal?

Answer 1

The Windows file cache will try to keep as much of a file in RAM as possible. In Windows NT, loading a very big file could trigger a bug. If the file didn't fit into the cache, it would never be released anymore forcing everything else out of RAM into the swapfile until your machine would constantly swap.

But your case is more simple. Opening the file in the viewer will load the file in the cache. Since you're still writing it, the cache item will grow accordingly. Now you exit the viewer but you're still writing. What should Windows do? Apparently, it keeps the file in the cache.

This is not a memory leak unless the cache never shrinks. Does it shrink a) when you run out of RAM or b) when you close the file in the writing process?

Answer 2

When my system run out of RAM cache doesnt shrinks, system unload everything into the swap file. When i close the file in the writing process, system cache shrinks.

Answer 3

If I were you I'd try and get a better understanding of what makes up the 'system cache' number which task-manager displays.

I suspect it's not what you're thinking, and your use of a phrase like "system unload everything into the swap file" particularly makes me wonder if you're trying to transplant experience on a different system onto Windows.

The Mark Russonovich books are considered to be the best reference for this sort of stuff.