How can I save a very large MATLAB sparse matrix to a text file?

Question

I have a 30000x14000 sparse matrix in MATLAB, which I need to use in another program. Calling save won't write this as ASCII (not supported). Calling full() on this monster results in an Out of Memory error. How do I export it?

I use MATLAB 7.

Answer 1

Did you try partitioning it ?

I mean try calling full() on the 1000 first rows (or 5000) and then repeat the process if it works.

Answer 2

Save the sparse matrix as a .mat file. Then, in the other program, use a suitable library to read the .mat file.

For instance, if the other program is written in Python, you can use the scipy.io.mio.loadmat function, which supports sparse arrays and gives you a sparse numpy matrix.

Answer 3

If this is pretty much a one time deal, then I would just iterate through the matrix and write the matrix to an ASCII file by brute force, or else use @Veynom's suggestion and call full() on a subset of rows. It may take a while, but it will probably be done faster than it might take to learn how to read in a .mat file outside of the MATLAB environment.

If this is something you need to do on a recurring basis, then I would take @Vebjorn's advice and use a library to read the .mat file.

Answer 4

I saved it as text using Java within MATLAB. This is the coolest thing I have ever seen in a scripting environment.

MATLAB Code:

pw=java.io.PrintWriter(java.io.FileWriter('c:\\retail.txt'));
line=num2str(0:size(data,2)-1);
pw.println(line);
for index=1:length(data)
    disp(index);
    line=num2str(full(data(index,:)));
    pw.println(line);
end
pw.flush();
pw.close();

Here data is an extremely large sparse matrix.

Answer 5

Use the find function to get the indices of non-zero elements...

idcs = find(data);
vals = data(idcs);
...save the index vector and value vector in whatever format you want...

If you want, you can use ind2sub to convert the linear indices to row, column subscripts.

If you need to recreate a sparse matrix in matlab from subscripts + values, use spconvert.

Answer 6

Midhat: How load this file using java?

Answer 7

You can use find to get index & value vectors:

[i,j,val] = find(data)
save_data = [i,j,val]

You can recreate data from save_data with spconvert, which is meant to "Import from sparse matrix external format" (so I guess it's a good export format):

data = spconvert( save_data )

You can save to ascii with:

save -ascii data.txt save_data

But this dumps indices as double, you can write it out more nicely with fopen/fprintf/fclose:

fid = fopen('data.txt','w')
fprintf( fid,'%d %d %f\n', save_data' )
fclose(fid)

Hope this helps.

Answer 8

dlmwrite - Write matrix to ASCII-delimited file Syntax

dlmwrite(filename, M)

dlmwrite(filename, M, 'D')

dlmwrite(filename, M, 'D', R, C)

dlmwrite(filename, M, 'attrib1', value1, 'attrib2', value2, ...)

dlmwrite(filename, M, '-append')

dlmwrite(filename, M, '-append', attribute-value list)