how to create a very large matrix using matlab

Question

Hello,

I have a problem with matlab when I'm trying to create a a matrix with a very large size such as 40000x40000.

for example:

x=zeros(40000,40000);

the error message is "??? Maximum variable size allowed by the program is exceeded. "

is there any solution.

Also I have another question, can we have a matrix with variable column size such as in java.

Answer 1

40k*40k => 1.6*10^9 Numbers. I believe Zeros will return floats. So you would need around 1.6*10^9 Floats *4 Bytes/Float => 6.4GB of RAM just for this one Variable...

Are you shure that you need such a big array?

Answer 2

40000 * 40000 * 8 bytes per number = 12 GB, surely you won't have enough memory.

To create a huge matrix with lots of zeros, you need a sparse matrix:

m = sparse(40000, 40000)

---

To create an array of variants, you can use a cell array:

m = cell(3, 1)
m(1) = [1,2,3]
m(2) = [2,4,6,8,10]
m(3) = 6+6i

Answer 3

no, well yeah buy more ram, as pointed out below. Sparse just removes zero elements.

And in matlab you can just append the new col/rows to the matrix to make a bigger matrix.

Ah but you mean, can you have a matrix where one column is 10 rows, and another is 20 for example. No, you can't do that. Didn't think you could in Java tbh.

Answer 4

If you really need a matrix that large, you could use the Parallel Computing Toolbox (and the MATLAB Distributed Computing Server) to harness the memory of several machines simultaneously. This would allow you to write:

matlabpool open <a large number>
x = distributed.zeros( 40000, 40000 );

See also: doc for codistributed arrays

Answer 5

There are many things you can do, as the various answers point out. The correct answer depends on your actual problem. Brute force creation of such a large array is impossible without a 64 bit version of matlab plus enough memory to store that large an array, or storing the array in some other way. You can store the array offline, only bringing in what you need as you use it.

For example, if these numbers will always be small integers, then use uint8 or int8, or a logical array, even a single array, all of which will reduce the memory requirements compared to double arrays. Or use a sparse storage form as appropriate. Much depends on what you will do with the array.

As far as a variable number of elements in a row, use a cell array here, not a flat traditional array.

Answer 6

I think use hard disk files to store that data is best choice

Answer 7

Aw... I came here hoping for better answers. I'm up against the same problem. I'm eliminating loops in my script by replacing those operations with matrix algebra, but that requires several matricies that are 117056x117056x3 which comes to about 330 GB each. Anyone know of a decent alternative to both loops and matrix math? My next guess is to use loops AND matricies to break the giant ones into smaller peices, but that will be slower, defeating the purpose of eliminating the loops at all...