MATLAB: maximum pre-allocated size?

Question

Hello there,

I'm writing a MATLAB program that will generate a matrix with 1 million rows and an unknown amount of columns (at max 1 million).

I tried pre-allocating this matrix:

a=zeros(1000000,1000000)

but I received the error that the "Maximum variable size allowed by the program is exceeded."

I have a feeling that not pre-allocating this matrix will seriously slow the code down.

This made me curious: what is the maximum pre-allocation size in MATLAB?

Update

Thanks guys for the insightful answers.

I'm a novice at Matlab and programming altogether, so I guess I never really thought about what it means to create an array of one trillion elements ;)

I'm going to look into sparse matrices, because the result I am aiming for in this particular problem will be a matrix consisting for the larger part of zeros.

Thanks again for your help, I'm very impressed by the level of knowledge here.

-Pieter

Answer 1

Take a look at this page, it lists the maximum sizes: Max sizes

It looks to be on the order of a few hundred million. Note that the matrix you're trying to create here is: 10e6 * 10e6 = 10e12 elements. This is many orders of magnitude greater than the max sizes provided and you also do not have that much RAM on your system.

My suggestion is to look into a different algorithm for what you are trying to accomplish.

Answer 2

I've been saying this a lot recently - we have been spoiled by our computers. We sometimes think they are infinitely large, infinitely fast. With "giga-" as a prefix in front of everything, we forget that they have limits. Here consider the amount of memory that you want to preallocate. 1e6x1e6 = 1e12 elements. Don't forget that you are assigning doubles here, so that is 8e12 bytes of RAM. Do you really have 8 terabytes of RAM? I wish I had your expense account, even if just for a day.

Worse, don't forget that some of the time you do something with a matrix, MATLAB might need to copy the matrix in memory. So you generally need at least twice that much memory. And as for the OS, don't forget that windoze is a notorious memory hog. ;-) I'll bet that 8 terabytes or so of RAM would keep it happy though.

Even worse, MATLAB needs to find contiguous memory for all of its arrays. So it needs to find a block of contiguous memory of that size.

While it is indeed possible to address that much memory on a 64 bit version of MATLAB, this just lets us get spoiled too. The point of all of this is to think about what you are trying to do, rather than just blindly trying to allocate an array of zeros that would probably stress the largest supercomputers made in the world today. Don't think about this in terms of the maximum size array that matlab will let you allocate. Recognize that your computer has limits, and keep your problems inside the practical limits of that computer. Unless of course, you really do do have 8 TB memory chips plugged into the sockets of your PC.

Answer 3

To find out the real maximum array size (Windows only), use the command user = memory. user.maxPossibleArrayBytes shows how many bytes of contiguous RAM are free. Divide that by the number of bytes per element of your array (8 for doubles) and you know the max number of elements you can preallocate.

Note that as woodchips said, Matlab may have to copy your array (if you pass by value to a subfunction, for example). In my experience 75% of the max possible array is usually available multiple times.