Dealing with excessively large matrices generated by ndgrid

3 views (last 30 days)
I need to run a function on all possible combinations of the elements of 5 vectors. Each vector is generated using linspace(x1,x2,n). The accuracy of the operation I'm trying to complete is increased if I avoid large differences between successive elements in each vector, so I would like n to be reasonably large. With this all in mind, I've written the following code:
n=1000;
p=zeros(6,n^5)
prep1=linspace(.1,1,n);
prep2=linspace(.1,1,n);
prep3=linspace(.5,5000,n);
prep4=linspace(1,10000,n);
prep5=linspace(.01,1,n);
sets = {prep1, prep2, prep3, prep4, prep5};
[p1, p2, p3, p4, p5] = ndgrid(sets{:});
prept = [p1(:) p2(:) p3(:) p4(:) p5(:)];
p=prept';
Where the first five rows of p are the elements of the first five vectors. The sixth row will be the output of the function evaluated using the data from the other rows within that column.
Using this method, I get a matrix of dimensions 6 by n^5, because I have 5 vectors, each with n elements, and the number of possible combinations of five vectors with n elements each is n^5.
This matrix is far too large for my system. I can try to get access to a better computer, but that may not be possible. I need some way to restrict the size of the matrix p so that I only deal with a chunk of it a time, save the values of my function evaluated for that chunk, and then move on to the next one. With this in mind, I wrote the following code:
clear all
tn=1000;
n=10;
p=zeros(6,n^5);
prep1l=linspace(.01,1,tn);
prep2l=linspace(.01,1,tn);
prep3l=linspace(.5,5000,tn);
prep4l=linspace(1,10000,tn);
prep5l=linspace(.01,1,tn);
for i=1:tn/n
prep1=linspace(.01,prep1l(n*i),n);
prep2=linspace(.01,prep2l(n*i),n);
prep3=linspace(.5,prep3l(n*i),n);
prep4=linspace(1,prep4l(n*i),n);
prep5=linspace(.01,prep5l(n*i),n);
sets = {prep1, prep2, prep3, prep4, prep5};
[p1, p2, p3, p4, p5] = ndgrid(sets{:});
prept = [p1(:) p2(:) p3(:) p4(:) p5(:)];
p=prept';
for d=1:length(p)
(...Function Code Truncated for Brevity...)
end
filename = ['p' num2str(i) '.mat'];
save(filename,'p')
end
This code takes the initial vectors, splits them into sub groups of 10 elements, and runs ndgrid on those subgroups. The problem is that this doesn't capture all of the possible combinations. Suppose that I had initial vectors of 10 elements and subdivided into sub-vectors of two elements, and then found the combinations. I would end up with 2^5, five times since 10/2=5, so I would have 5 p files (p1, 2, 3, 4, 5) each of length 2^5, or a total length of 5*5^2, which is significantly smaller than the real number of combinations, 10^5. So, this method does not work.
I need some way to tell matlab to begin using ndgrid, but stop after a certain number of columns so that I don't run out of memory. Evaluate the function according to the second nested loop, save p to a file, and then repeat, calculated the next batch of values to be worked upon. Is this possible?

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!