Windows Vista+ Memory Problem with IFileOpenDialog and large stack

Question

I have experienced a strange problem in windows vista and above. When I use the IFileOpenDialog with a large stack, the amount of memory remaining after showing the dialog drops by about a gigabyte.

#include <windows.h>
#include <Shobjidl.h>
#include <iostream>

void memGrab(char *);
int main(int argc, char **argv){
  char Message[128];
  CoInitialize(NULL);
  if(argc > 1){
    wsprintf(Message, "Starting Memory Test: %c", argv[1][0]);
    std::cout << Message << "\n";
    if(argv[1][0] == 'b')
      memGrab("before");
    TCHAR *fileName = (TCHAR *)malloc(sizeof(TCHAR) * 1024);

      IFileOpenDialog *pfd;
    int index = 0;
    int len = 0;
    int i = 0;

    HRESULT hr = CoCreateInstance(CLSID_FileOpenDialog,
            NULL,
            CLSCTX_INPROC_SERVER,
            IID_PPV_ARGS(&pfd));

    if(SUCCEEDED(hr)){
      pfd->Show(NULL);
    }
    pfd->Release();
    if(argv[1][0] == 'a')
      memGrab("After");
  }
  CoUninitialize();
}

void memGrab(char *text){
  for(int i = 0; i < 400000; i++){
    if(!malloc(10240)){
      char Message[128];
      wsprintf(Message, "Memory Gone %s: %d K", text, i * 10);
      std::cout << Message << "\n";
      exit(0);
    }
  }
}

When I compile this program without setting a stack size (cl testMem.cpp Shell32.lib Ole32.lib user32.lib). I get the following result:

C:\RWSource>testMem.exe b
Starting Memory Test: b
Memory Gone before: 1984040 K

C:\RWSource>testMem.exe a
Starting Memory Test: a
Memory Gone After: 1875640 K

However, when I compile it with setting the stack size (cl testMem.cpp Shell32.lib Ole32.lib user32.lib /F100000000). I loose a large amount of allocatable memory.

C:\RWSource>testMem.exe b
Starting Memory Test: b
Memory Gone before: 1795370 K

C:\RWSource>testMem.exe a
Starting Memory Test: a
Memory Gone After: 463840 K

Update 1:

I checked the memory with VMMap (thank you TheUndeadFish) and when I open the dialog, there are multiple threads started that all have about 100MB. Is there any way to make the main thread have a large stack while not giving the child threads 100MB each?

Answer 1

I don't know exactly what's going on in your scenario. But I have seen the File Open Dialog cause other memory trouble before.

When that dialog is invoked, a lot of things may get loaded into your process. In particular, shell extensions that appear in Windows Explorer also get loaded in with the File Open Dialog. That's how you're able to right click on a file in that dialog and get all those non-standard goodies. However, this means that the dlls that underlie those extensions are getting loaded into your process.

From experience, I have seen that some of those dlls do not get unloaded once the File Open Dialog is closed. Instead they stay present, taking up some of the address space of your process. (Maybe this is some kind of bug on the part of those dlls, or maybe it's some kind of "optimization" to keep them loaded in case the File Open Dialog is used again.)

The consequence of this that I dealt with was that it affected the success of a single 1GB memory allocation. Before opening the dialog there was a large stretch of address space available and a 1GB allocation could succeed. However after opening the dialog, a couple of the shell extension dlls landed right in the middle of the available address space and divided it into chunks smaller than 1GB, thus causing the allocation to fail.

I know this anecdote doesn't relate directly to what you're doing, since you're allocating many small amounts instead of one large amount. (And in my scenario it wasn't malloc being used. I think it was something like VirtualAlloc or such.) But I would guess something roughly similar may be happening.

The File Open Dialog may be loading something into your process that somehow inserts a barrier into the space that malloc would normally use. But somehow this only happens when you're using that large stack size because I guess that extra 99MB of space reserved for the stack is somehow rearranging the rest of the address space to be susceptible to this problem.

When I was investigating my own issue, I used VMMap to take snapshots of my process's address space at various points to figure out what was happening. You might have some luck doing that. Just note that it looks at the address space like the operating system sees it, not really like you see it from inside the context a specific language like C++. So it's not necessarily easy to figure out what particular part of memory is being used by malloc or any other allocation mechanism.