Why does my file reading operation using FGETL operation take so long?

I am having trouble reading in a large text file (greater than 200,000 lines). I am using low-level file IO via the fgetl() function. The read loop just goes away and does not finish after hours.

In order to speed up the reading, I attempted to preallocate a character array into which I read the lines of the file. I know something about the width of the lines in the file and basically allocate more space than I'll need. Based on the fact that I already know the width and using the file size of the file, I attempt to get the number of lines present in the file.

With the pre-allocation change it still ran for a long time and made MATLAB hang.

Answer 1

[MathWorks Support Team]

In order to speed up the file reading process, reallocation of the data should be avoided. Reallocation occurs when a regular array is resized, and it will slow down the read process.

You can do some preallocation by determining the number of lines in the file. To obtain the number of lines use FGETL to run through the file and count the number of lines.

Once the number of lines is known, a cell array can be created. When a new line is stored in the cell array, only the contents of the single cell is allocated. That will speed up the reading process.

The following code demonstrates this technique:

num_lines = 1;

while feof(fid) ==0;

      fgetl(fid);

      num_lines = num_lines +1;

end

fclose(fid);

% assign length as the number of lines to be read.

length = num_lines;

datafile = cell(length,1)        %preallocation