How to allocate more memory to your program( GCC)

Question

Hi,

I want to allocate more memory to program.l What is the gcc flag that allows you to do so?

FYI what I am trying to do is create a very large matrix( really large) which is gonna go through compression algorithms later. So there is no way I can avoid creating such a large matrix to store data.

Answer 1

Use the heap! malloc() and friends are your friends.

Answer 2

If the matrix is really large you might have to allocate memory in smaller segments so it can find room in the virtual memory space. On 32-bit Windows I have found you simply cannot get anything bigger than about 980 MB in a single allocation. On Linux it is pushing it to try to get more than about 1.5 GB.

In a 64-bit system you can get a lot more.

But in any case, I would recommend using a matrix library that can handle the memory and algorithms for you. There are many subtle tricks to making fast matrix computations. Tricks with threads, computing in cache-sized blocks, prefetching data, SSE vector ops, etc.

You might want to look into using the math libraries from either Intel or AMD.

Answer 3

You don't need any special gcc flags.

Use malloc to allocate your array dynamically at runtime.

If you are somehow forced to use a static array, or if your environment is set up by default to limit your program's access to virtual memory, you may need to use the ulimit command.

ulimit -v unlimited
ulimit -d unlimited

Otherwise, you need to specify more clearly the error you are getting that prevents you from getting sufficient memory, and probably also tell us how big your matrix is.

Answer 4

Your question is very unclear, but I suspect that you are trying to create a large multidimensional array (matrix) as a local variable (auto variable) to some function (possibly main) and this is failing.

int foo(int boo, int doo) {
    int big_array[REALLY_BIG];
    ...

This would fail because C compilers try to make room for variables like this on the programs system stack. A compiler may just fail upon attempting to think about something that big being on the stack (especially with alignment issues that might make it bigger) or it may generate code to try to do this and either the CPU can't run it because stack pointer relative indexing is limited or because the OS has placed limits on the size of the program's system stack.

There may be ways to change OS limits, but if it is a CPU limit you are just going to have to do things differently.

For some things the simplest thing to do use just use global or static variables for large sized data such as this. Doing this you end up allocating the space for the data either at compile time or at program load time (just prior to run time), but limits your ability to have more than one copy since you have to plan ahead to declare enough global variables to hold everything you want to be live at the same time.

You could also try using malloc or calloc to allocate the memory for you.

A third option is (if you are using a *nix system) to memory map a file containing the matrix. Look into the mmap system call for this.

An added benefit of using mmap or static or global variables is that under most operating systems the virtual memory manager can use the original file (the file containing the matrix for mmap, or the executable file for static or global) as swap space for the memory that the data uses. This makes it so that your program may be able to run without putting too much of a strain on the physical memory or virtual memory manager.