How to troubleshoot an 'Out of memory' error with pca() when matlab still has plenty of RAM free?

Could you help troubleshoot an 'Out of memory' error when running pca() on a 6000 x 9500 matrix of doubles?

To reproduce the error, run:

pca(rand(6000, 9500));

This produces the error:

Error using svd
Out of memory. Type "help memory" for your options.
Error in pca>localSVD (line 484)
[U,sigma,coeff] = svd(x, 'econ');

More specifically, pca will work on 6000x5000 but not an 6000x6000 matrix.

There should not be an 'Out of memory' error here because:

1. matlab still has plenty of RAM available: this system has 32 GB of RAM and the matlab process nevers consumes more than 15% of that, even during pca (according to the 'top' command in ubuntu).
2. R does PCA on that matrix without problem ;-)

Operating systems limits don't seem to be the solution:

A. running 'ulimit' on the command line suggests no memory limits:

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       126664
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  16384
-v: address space (kbytes)          12582912
-x: file locks                      unlimited
-i: pending signals                 126664
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

B. Matlab is running in a 64-bit environment: ubuntu 18 LTS 64 bit with Matlab R2019b. The 'computer' matab command returns 'GLNXA64' so I believe I am using the 64-bit version of matlab. The 'ver' command also reports an x86_64 architecture and a 64-bit java hotspot server.

This question has been asked before. But it seems relevant to ask it again because all previous questions refered to the old implementaiton of pca (princomp()), are for systems with significantly less RAM (8 GB), for 32 bit environments, or were caused by issues in the data matrix (such as NaNs filling one column).

[Jason Ross]

"ulimit -a" gives more useful output -- if you could edit your question and provide that output it would be more helpful.

The last release of MATLAB to support 32-bit Linux was R2012a (https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/support/sysreq/files/SystemRequirements-Release2012b_Linux.pdf). So you are most definitely running 64-bit Linux.

[dpb]

R2019b had no problem here on a Win7 box w/ 8 GB physcial memory

[Jean Hausser]

Thanks for checking dpb. That points away from a pure matlab issue and towards an issue at the interface between matlab and the operating system.

Thanks for the feedback Jason. I have now edited the question to provide the output of ulimit -a. What do you think we should check next?

[the cyclist]

Just wanted to share another data point:

tic; pca(rand(6000, 9500));toc % Your original size
Elapsed time is 78.365737 seconds.

tic; pca(rand(6000, 19000));toc % Double your 2nd dimension
Elapsed time is 95.615801 seconds.

tic; pca(rand(12000, 19000));toc % Double both dimensions
Elapsed time is 731.404131 seconds.

R2019b on fairly new MacBook Pro with 16GB.

[Jean Hausser]

New observation: pca(rand(6000, 9500)) actually runs fine in a fresh Matlab environment here, when Matlab is just started.

pca(rand(6000, 9500)) only fails if I first load the data I need for that PCA. But I still don't get why: before starting pca(rand(6000, 9500)), Matlab just uses 8.8% of RAM (VIRT 10G, RES 2.7G, SHR 400M). Memory never climbs higher than 15% before PCA throws the 'out of memory error', so Matlab has plenty of RAM.

[dpb]

Insufficient contiguous memory then is likely the culprit...

[Jean Hausser]

dpb: I thought of that too, but how could it possibly be? Take a look at how mem usage changes as I prepare for pca:

Action		memory usage
start matlab	4%
load data	8%
precompute 	9% then back to 8%
drop bad data	7.5%

So even the first 9% of memory are unusable due to memory fragmentation, Matlab still has 91% of RAM available to play with.

Or does matlab maybe store objects at random places in the RAM instead of stacking it up at the first available address? This could increase fragmentation. But even so, my intuition would be that there is tons of contiguous RAM to play with.

What do you think? How can we check if memory fragmentation is the culprit?

[Hari Krishna Ravuri]

I failed to reproduce this behaviour on my end. Please provide reproduction steps.