How can I sum a large (10000x10000) matrix quickly?

Hi MATLAB community,

I am trying to sum a 10000x10000 matrix of values +1, 0, and -1, and it is taking a longgg time.

I have tried both

term = sum(sum(A));

and

term = sum(A(:));

I do not get an error message, my program just gets hung up on this line of the code and never completes.

Any suggestions on an efficient way to sum all elements in this matrix?

Thanks!

Answer 1

[Walter Roberson]

It should be fast:

>> ABC = randi([-1 1], 1000,1000);
>> timeit(@() sum(ABC(:)))
ans =
           0.0003416905955

and that was about the slowest.

I also find I can speed it up marginally by having used an array of int8:

DEF = int8(ABC);
sum(DEF(:))

seems to be faster on average, but not always faster. On the other hand the time required do do the int8() in the first place is consistently longer than the time required to do the sum(), so I would not bother. (It might make sense to do this on much large matrices, in which case you would want to do it at the time the data was read in -- which could be done for textscan() but not so much for xlsread())

[declan]

Thanks for the "timeit" hint -- I timed the sum of 10,000 x 10,000 matrix, then calculated how long it would take to do that 10,000 times... turns out I just need to be more patient!

[Walter Roberson]

On my system, matrices of 10000 by 10000, done 10000 times, would take about 6 minutes.

Answer 2

[John D'Errico]

Actually, it DOES complete, or it will, eventually. It is just doing some heavy disk thrashing, swapping deeply into virtual memory. So your hard drive is probably working like mad there. Eventually it will come back to life.

Your problem is you are trying to solve too large of a problem for the memory that you have. So either you need to get more memory, or you need to work with smaller problems. Depending on how many zeros in the matrix, you might gain some by use of sparse matrices. (Don't even bother with sparse unless your matrix is at least 90% zeros. In most cases, it should be way more sparse than that, but if all you are doing are sums, 90% zeros should show you a gain.)

[declan]

Yes, I knew the number of "steps" assigned (10,000) for the calculation was bogging it down, but unfortunately I don't have any other option! Guess I'll just wait it out!

[Walter Roberson]

A 10000 by 10000 matrix would be about 800 megabytes, which should not cause swapping on its own unless you were on a system on the low end of MATLAB's specifications of several years ago.