Exponentially slow save rate with -v6

1 view (last 30 days)
Hi, I am trying to save a hours worth of recording that has been split up into sections of 5,500 frames (uint8) which is under the 2GB cap that the -v6 is limited too. Saving these groups of 5,500 as their own .mat file (yes I know not the best way to do this), is pretty fast in the command window. Ran 20 times in a for loop avg. time was around 18 seconds per group. However, the script I wrote, using the same exact commands to save 20 groups it takes anywhere from 25 - 66 minutes which is not optimal or my research. Any reasons why the save gums up?
  3 Comments
Geoff Hayes
Geoff Hayes on 26 Jul 2014
Nicholas - perhaps attach some (or all) of the code that is in your script so we can see what it is doing.
Nicholas Vitale
Nicholas Vitale on 28 Jul 2014
for i = 1:maxCycle
dummyCel{i} = savedFrames{i};
save(sprintf('C:/Users/vitale/Documents/MATLAB/Recordings/%s/FramePackage%d....mat,cyclename,i'),'dummyCell','-v6');
savedFrames{i} = [];
end
To clarify the variables: savedFrames is a cell array within a cell array. It has 30 slots that each hold 5,500 frames each. I had to do this since I couldnt add a part of the mother cell array to the workspace dynamically.
dummyCell is just a holder. I set the part I want of savedFrames to it. Save dummyCell and delete the data in savedFrames and overwrite dummyCell on the next iteration.
maxCycle is just how many times I want to go through my for loop.
I also save each part of the frames in a different folder since I couldnt find a I way to append all the frame packages together in one .mat file quickly or efficiently.
Let me know if more code is needed.

Sign in to comment.

Accepted Answer

Geoff Hayes
Geoff Hayes on 28 Jul 2014
To be clear - savedFrames is a cell array of 5500 frames, and is one of 30 slots within a greater cell array. Does this mean that maxCycle is 30?
On each iteration of the for loop, the code copies the 5500 frames from the ith slot of savedFrames into the placeholder dummyCel
dummyCel{i} = savedFrames{i};
Note that dummyCel is being treated as a cell array and that its ith slot is being updated at each iteration - so it is growing at each iteration of the loop! So not only is the code writing out the 5500 frames (to the mat file) from the current iteration, but it is including the 5500 frames from each of the previous iterations as well. So each mat file is getting larger and larger, and the time to write out each is taking longer and longer.
I think that you want to replace your above code with just the following (or something like this, I truncated the name to work on my computer)
for k = 1:maxCycle
dummyCel = savedFrames{k};
save(sprintf('FramePackage%d.mat',k),'dummyCel','-v6');
end
There is no need to clear the ith slot of savedFrames. Note that all files will have the same named variable in each, dummyCel. Unless, by using dummyCel{i}, were you trying to give each a distinct name?
NOTE that in the above code, I replaced i with k since i and j are used by MATLAB as representations for the imaginary number.
  1 Comment
Nicholas Vitale
Nicholas Vitale on 28 Jul 2014
Thanks for the feedback. I didnt realize I was doing this. Just one of those days where your mind doesnt catch the extremely stupid mistakes. Thanks for pointing out my error!

Sign in to comment.

More Answers (0)

Categories

Find more on Loops and Conditional Statements in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!