HELP running large code.. having storage issues

I am attempting to sort through csv data points in order to filter a certain data type out to put into a new csv. The large csv contains 55 columns and approximately 43 million rows. I am running into storage issues from Matlab and my computer, even though I have a 500GB adidtional storage drive that I am working through. It takes about 4 1/2 hours to run this code, but I always seem to have an error when I check back. It is not a java heap error, but just states "out of memory. Type HELP MEMORY" and I get a popup box that says low on storage and shuts matlab down. This is haulting my project and I am not sure where to go from here. Any recommendations are appreciated...
clc;clear;
myfile = fopen('file_2009.csv');
whichSite = input('Which site? ');
year = 2009;
recs{10e4,1} = [];
ctr = 1;
while ~feof(myfile)
linestring = fgetl(myfile);
supercell=strsplit(linestring, ',','CollapseDelimiters',false);
if str2double(supercell{3}(2:5)) == whichSite
recs{ctr,1} = supercell;
ctr = ctr+1;
end
end
fclose(myfile);
mycsvfile = sprintf('Site%d_%d.csv',whichSite, year);
recstable = cell2table(recs);
writetable(recstable,mycsvfile);
recs = [];
recstable = [];

1 Comment

Surely you can just read and write your data in sensible-sized chunks instead of trying to store 10^4 cells in memory at once? It seems to just be a sequential operation so I would assume you can just append results to those previously saved.

Sign in to comment.

Answers (0)

Categories

Find more on Data Import and Analysis in Help Center and File Exchange

Products

Asked:

on 20 Feb 2019

Commented:

on 20 Feb 2019

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!