How do I load a LARGE file into a DISTRIBUTED ARRAY on a CLUSTER?
1 view (last 30 days)
Show older comments
I have a huge binary file, twoHundered.bin, . It isso huge that you could not feed it into the memory of one machine, but it would fit into a four node cluster. This file is flat uint8. Put its contents into a distributed array. You are not allowed to read it into one node, and then distribute it (because it would not fit onto one node).
Accepted Answer
MathWorks Support Team
on 27 Jun 2009
You can use the memmapfile to help you load a large file in a distributed array.
P>> m = memmapfile('twoHundered.bin');
P>> V = dcolon(1,50); % 50 is number data elements
P>> data = m.Data(local(V));
P>> data_distrib = darray(data);
0 Comments
More Answers (0)
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!