Handling large amount of structures in MATLAB

Question

I need to handle massive (tens of millions) of MATLAB structs; I needed a dozen or so fields so I reckoned memory won't be an issue, until I discovered this ( explanation )

>> s=[];
>> s.first=1;
>> whos
  Name      Size            Bytes  Class     Attributes

  s         1x1               132  struct              

>> s.second=2;
>> whos
  Name      Size            Bytes  Class     Attributes

  s         1x1               264  struct              

>> s.third=3;
>> whos
  Name      Size            Bytes  Class     Attributes

  s         1x1               396  struct

Which obviously stops me from using tens of millions of much larger structs.

Resorting to classes solves the memory usage problem (a markup of 56 bytes per struct array) but it is prohibitively slower on the construction and in the destruction of objects.

How can I create stuct which is lightweight (like C structs) and fast?

Answer 1

What do you mean with classes? As far as I remember classes was the term in matlab for type. I guess you mean a self defined class.

A solution (which is also recommended in to the matlab docs) is to switch from an array of structs to an struct of arrays (look in your link at the R, G, B) example.

Answer 2

Convert these structs into arrays, and then provide accessor methods via a class.

Answer 3

(a) use big arrays (where the 'first' field of struct 1 is element 1 of the 'first' array, for struct 2 it's in element 2, etc.), as Pyrolistical suggests.

(b) consider using another language like C++ (or maybe Java) that provides better control over memory usage. You can access the C++ code via mex functions (which can a little difficult some times). You can evaluate Java bytecode directly from Matlab.

Answer 4

Another option: hold the struct of arrays in a hidden global struct. Create a class object that digs into this global struct to slice out the data that apply to just that one instance.

The global struct can be implemented more cleanly with the PERSISTENT keyword and/or using private/ directories for information hiding. If you're using 2008a or newer, the new handle object system should help make the implementation much cleaner.

If you really have large and complex data structures, I'd seriously consider another language like C++, Java, or Python w/ numpy. I love Matlab when my tasks map well to it. Data structures are not one of Matlab's strong points, especially pre-2008a.