How to create variable in MAT file that is too large to fit in memory?

The new 2011b release provides the matfile object, allowing you to load a portion of a large MAT-file variable using indices. This is useful if the entire variable cannot fit in memory.

But my question is, how can I create such a large variable in a MAT file in the first place, if I cannot construct it entirely in memory first? The save -append option only adds variables to a file; it does not append elements to a variable in the file.

I see that the matfile object allows for partial save (by indices) in the same manner as partial read, so if I could somehow allocate such a large variable (like a zeros() for matfile), then I could write to it piecemeal.

[Oleg Komarov]

Not such a great addition then...

Answer 1

[Jeff]

So, I finally got the answer I was looking for at an on-campus MATLAB seminar not too long ago. If I have a matfile object, matObj, then preallocating a large variable is as simple as:

matObj.myVar(1000:1000) = 0;

I also just googled and found a blog post by Loren Shure that demonstrates this: http://blogs.mathworks.com/loren/2011/10/14/new-mat-file-functionality-in-r2011b/#17

[NbrOneProcastinator]

Although this thread is already old I have to comment on this answer, which is actually not quite right. To preallocate memory for a large matfile you have to define the last element, e.g.

matObj.myVar(1000,1000) = 0;

Notice the comma instead of the colon.

Answer 2

[James Tursa]

You could create the MAT file from scratch in a Fortran or C/C++ program using the published uncompressed MAT file format. E.g., here:

http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf

A variable of the desired size doesn't have to exist in memory in order for the MAT file to be written.

[Jeff]

Certainly, but that is far from ideal. It seems this new feature is without a useful analog.

Answer 3

[the cyclist]

One simple possibility is that the large array was created and saved when it was the only variable in the workspace, but later is loaded when the workspace is littered with other variables, so there is only room for part of it.

[Jeff]

That's a good point, but I think I should have phrased this less like a hypothetical question. I am specifically interested in creating such a large MAT-file variable, say, as I read it in blocks from a binary or ASCII file.

[Walter Roberson]

Or the large array was constructed on a system which had more memory, including possibly a 64 bit MATLAB version for a file being loaded on a 32 bit MATLAB version.

Subnote: if you are planning to use such large matrices, you should probably be saving with -v7.3 as your files might be too large for -v7 . But beware that -v7.3 file performance can be much worse than -v7 file performance (unless they fixed that.)