Can I run out of virtual memory on linux?

Question

My application similar to hypotetical program:

for(;;) {
  for (i=0; i<1000; i++) {
    p[i] = malloc(random_number_between_1000_and_100000());
    p[i][0]=0;  // update
   }
  for (i=0; i<1000; i++) {
    free(p[i]);
  }
}

Has no memory leaks but on my system, the consumption of memory (top, column VSS) grows without limits (such as to 300% of available physical memory). Is this normal?

Updated - use the memory for a while and then free it. Is this a difference?

Answer 1

The behavior is normal. Quoting man 3 malloc:

BUGS

By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. This is a really bad bug. In case it turns out that the system is out of memory, one or more processes will be killed by the infamous OOM killer. In case Linux is employed under circumstances where it would be less desirable to suddenly lose some randomly picked processes, and moreover the kernel version is sufficiently recent, one can switch off this overcommitting behavior using a command like:

       # echo 2 > /proc/sys/vm/overcommit_memory

See also the kernel Documentation directory, files vm/overcommit-accounting and sysctl/vm.txt.

You need to touch (read/write) the memory for the Linux kernel to actually reserve it.

Answer 2

As long as you don't touch those allocated chunks the system will not really allocate them for you.
However you can run out of addressable space which is a limit the OS imposes to the processes, and is not necessarily the maximum you can address with the systems pointer type.

Answer 3

Try to add

  sbrk(-1);

at end of each loop to see if it makes difference.

The free() only deallocates memory but it doesn't give it back to OS.

Answer 4

The OS usually allocates all pages as Copy-On-Write clones of the "0" page, that is a fixed page filled with zeros. Reading from the pages will return 0 as expected. As long as you only read, all references go the the same physical memory. Once you write a value the "COW" is broken and a real, physical, page frame is allocated for you. This means that as long as you don't write to the memory you can keep allocating memory until the virtual memory address space runs out or your page table fills up all available memory.