malloc and free in D/Tango not freeing memory?

Question

here is a simple d/tango code in windows:

module d_test.d;

import tango.util.log.Trace;
import tango.core.Thread;
import tango.stdc.stdlib : malloc, free;

void main() {

    Trace.formatln("Checking in...");
    Thread.sleep(10);

    int total_n = (100 * 1000 * 1000) / int.sizeof; // fill mem with 100MB of ints
    int* armageddon = cast(int*)malloc(total_n * int.sizeof);

    for(int i = 0; i < total_n; ++i) {
     armageddon[i] = 5;
    }

    Trace.formatln("Checking in...");
    Thread.sleep(10);

    free(armageddon);
    armageddon = null;

    Trace.formatln("Checking in...");
    Thread.sleep(10);


}

when I run the program, memory stays low ~2MB, and when I allocate an array of 100MB to the pointer memory usage jumps to ~100MB, which is fine. However, after free memory is still (I'm looking at task manager) at 100MB to the end of the program.

I thought it might be to windows pagefile cache or something, so I've tried a simple C++ program:

#include <iostream>
#include <windows.h>

using namespace std;

int main() {

  Cout << "Checking in..." <<< endl;
  Sleep(10000);


  int total_n = (100 * 1000 * 1000) / sizeof(int);
  int* armageddon = (int*)malloc(total_n * sizeof(int));

  for(int i = 0; i < total_n; ++i) {
    armageddon[i] = 5;
  }

  Cout << "Checking in..." <<< endl;
  Sleep(10000);

  free(armageddon);
  armageddon = NULL;

  Cout << "Checking in..." <<< endl;
  Sleep(10000);


return 0;
}

I've compiled it with g++ and everything seems to work like it should. When program starts - memory usage ~900kb, after allocation ~100MB, after free ~1,2MB...

So, what am I doing wrong or is this a bug?

Answer 1

When you free memory with "free", it's likely that it doesn't actually free the memory.

It could be that it is simply marking it as free but holding on to it in case you request more memory later. If for some reason the system starts running low on memory, the runtime might actually free it at that point.

Answer 2

You can start by looking at _heapmin(). free() does not return unused heap to the OS, it it only marks it as free and coalesces with nearest free neighbors.

Answer 3

It is going to depend on how free is implemented. For some interesting reading, take a look at Doug Lea's allocator, which tries to be efficient at managing memory requests that can span a wide range of sizes. His primary concern is that the malloc and free calls be fast, and correct.

But he does mention the issue of returning memory to the OS, and discusses what hinders it (memory fragmentation) and what aids it (use of mmap, or in a less flexible manner, sbrk). Once you've read this article, you'll have a clearer idea of the trade-offs which lead to infrequent return of memory to the OS.

Answer 4

This is strange. From looking at the Tango source, malloc/free in tango.stdc.stdlib are just the C standard library functions, so there should rightfully be no difference - and in fact, when I try it using Phobos and std.c.stdlib under Linux, the memory goes right back down as it should.

Are you sure you're measuring it right?

PS: You can just do armageddon[0 .. total_n] = 5;

PS2: I tried your Tango code under Linux, and it goes back down as expected. Looks like a Windows issue.

Answer 5

It just means that Digital Mars' implementation of malloc and free doesn't return memory to the OS, even when you allocate a huge chunk like that. You could try using malloc and free from msvcrt.dll instead, maybe they'll free the memory.

Or you can use the Windows API directly, if this problem only exists for DMD on Windows. The easiest is to use the HeapAlloc and HeapFree functions, but I don't know if they'll do what you want either. A sure way is the more low-level VirtualAlloc and VirtualFree, which allocate directly from the OS instead of creating a heap first. Those are also the functions that HeapAlloc, malloc, new in C++, etc, ultimately use when they request memory from the OS.