standard deviation takes for ever

I have a double precision numeric 3D matrix M (converted by fread from uint8) of size 30000 x 500 x 500 I would like to get standard deviation along dimension 2 tic, std(M,0,2) ; toc has taken more than 12 hours and still running meanwhile mean(M,2) only took 80 seconds.

Or a bit more details.. std(M(:,:,1),0,2) takes 0.3 seconds and std(M(:,:,1:100),0,2) takes 34 seconds But std(M(:,:,1:500),0,2) says out of memory

Similarly mean(M(:,:,1),2) takes 0.1 seconds But mean(M(:,:,1:500),2) does not work and gives me 'out of memory' message But mean(M,2) takes about 80 seconds. This is all very confusing! Thanks

[NAVNEET NAYAN]

That might be occuring due to the size of 3D matrix and your system memory.

[gujax]

Hmm, so I should be really using a for-loop

When I do that I do not have memory issues. It does take long but about 200 seconds for standard deviation. That is way better than PC hanging up.

Wow. I did not appreciate subtle differences between

std(M,0,2) %PC hangs

std(M(:,:,1:500),0,2)%says out of memory

for j=1:500

Sdev(:,j)=std(M(:,:,j),0,2)

end

% takes 200 seconds

[Dyuman Joshi]

"std(M(:,:,1),0,2) takes 0.3 seconds and std(M(:,:,100),0,2) takes 34 seconds

But std(M(:,:,500),0,2) says out of memory"

This is peculiar. How are you running these timing tests?

[dpb]

30000*500*500*8/(1024*1024*1024)

ans = 55.8794

is an array of 56 GB; how much real memory does your machine have? The high run times are likely owing to disk paging referring to locations in the original array and working on the second dimension means not accessing memory in sequence but by steps of the size of the first dimension for each subsequent row.

You could try

M=M.';
std(M(:,:,100)).'

Speed should be better if did

std(squeeze(M(:,:,100)),0,2)

etc., ... but that may force a memory copy and cause memory issues, I don't know.

I don't have anything close to enough memory to even try...

[gujax]

Hmm did I make a mistake? Oh one mistake I made was to forget to write 1:100. Instead I wrote 100. I have edited my original query.

One file is an accumulation of (500 x 100x 5) files each 31 KB in size.

That comes close to 8 GB. Not 56 GB unless you meant bits?

[dpb]

Your original posting says "I have a double precision numeric 3D matrix M of size 30000 x 500 x 500..."

That's what I calculated above at 8 bytes/double takes up 59 GB storage.

I don't follow what " an accumulation of (500 x 100x 5) files each 31 KB in size." means?

Think you're going to have to show us specifically what your array is and how it was constructed.

[gujax]

Ah got it!

I append 100 x 500 x 500 times a 31 KB time series streaming data chunk into one file instead of generating 5 million separate write files.

So that’s about ~8GB data

But when I read it I didn’t quite realize by default fread converts it to double

Answer 1

[gujax]

calculating statistical std takes more memory than calculating mean. If performing std on double formatted large data sets, it likely will slow down the computer if memory is limited. That may not be true for evaluating statistical mean.

Answer 2

[Steven Lord]

Can you confirm you're using the std function included in MATLAB? What does this command show?

which -all std

/MATLAB/toolbox/matlab/datafun/std.m
/MATLAB/toolbox/matlab/datatypes/tabular/@tabular/std.m    % tabular method
/MATLAB/toolbox/matlab/datatypes/datetime/@datetime/std.m  % datetime method
/MATLAB/toolbox/matlab/datatypes/duration/@duration/std.m  % duration method
/MATLAB/toolbox/matlab/timeseries/@timeseries/std.m        % timeseries method
/MATLAB/toolbox/matlab/bigdata/@tall/std.m                 % tall method
/MATLAB/toolbox/parallel/parallel/@distributed/std.m       % distributed method

[gujax]

Yes I see all the above in response to your command. So how would I know which one I am using?

[Steven Lord]

You don't see any other std.m files in the list?

[dpb]

Presuming the answer to @Steven Lord's followup Q? is "No", then you've not accidentally aliased it as he was looking for (and wouldn't expect it likely to have done with the same input footprint as the original, but always worth checking), then for a double as input the base datafun one is the one that will be called; the others are overloaded versions for the specific data types/classes noted in the comments.

[gujax]

Sorry, other than what you showed I am getting @fints\std.m % fints method.

I missed that - didn’t look carefully.

I have an update..

I convert data to single instead of double and I do that in the read binary function call.

Now,

mean(M,2) takes 0.49 sec , a phenomenal reduction in time and I really don’t need double because my data is uint8.

With double it took 80 seconds.

However,

std(M(:,:,1:500),0,2) gives me out of memory error.

But in a for-loop as in my above posts, I get about 32 seconds.

Not bad at all though I thought going from double to single would only affect speed by 2x? .

However instead I see 10 fold improvement I am really wondering if I am doing something incorrect?

[Dyuman Joshi]

If your data is uint8 why not just use uint8? It will help with the memory, uint8 requires only 1 byte for storage, compared to 4 byte for single and 8 byte for double.

"I thought going from double to single would only affect speed by 2x?"

What makes you think so?

[gujax]

I would like to do float arithmetic on that uint8 data eg like standard deviation. I cannot on uint8 isn’t it?

[Dyuman Joshi]

Ah yes, Idk how I overlooked the obvious, My bad.

[gujax]

I think I will state this issue resolved? i.e., calculating statistical std takes more memory than calculating mean. If performing std on double formatted large data sets, it likely will slow down the computer if memory is limited. That may not be true for evaluating statistical mean.

[dpb]

The issue you're having must be in disk swapping owing to limited real memory...I'm still not positive about just how big your array is. How about

whos M

? to tell us precisely what you've processing and

memory

for the available memory your machine has?

It depends on how TMW builds the executable and what processor instructions they assume; unfortunately, it's likely they code to a "lower common denominator" of what is out there because know that not all customers are going to have latest CPU technology with enhanced vector processing instructions making use of builtin vector pipeline that exists with current processors.

I've never messed with trying it out, if you have a high-memory graphics card, you could possible try the GPU stuff...