fopen fails to open file at certain number of iterations

Hello all,

I have a GUIDE program that is using a loop to read data within .csv files that are contained within a folder. The file names are numbers i.e. 1.csv, 2.csv, 3.csv etc. Each file is about 5-6 MB and contains columns of numerical data.

At some point in the code there is an fopen to open and read in the relevant data from the files and this is assigned a file ID:

fID = fopen(strcat(fullfile( file location, file name),'.csv'),'r')

At a further point within the code, the file is closed and when the loop executes again the next file will be opened.

When i'm running the program on, for example, 100 files within the folder, whether they are 0.csv to 99.csv or 3400.csv to 3500.csv, the program executes fine with no issues, processes the files and outputs the desired information. However, when attempting to run the program with say 4000 files in the folder, the program halts but no error is shown in the command window.

Running

echo on all

allowed me to see that at some point in the code it flagged the error ' Error while evaluating uicontrol Callback'.

I was able to find that this error was occuring at the fopen line (as above).

The program was also stopping at the same file every time it was being run at 510.csv. To eliminate the cause of the issue being the source files, i deleted 50 files before 510.csv and ran it again. 510.csv processed fine but then it stopped at 560.csv, 50 files later.

I had a look at the fID values for the original run and it appears that the previous file 509.csv had a fID = 511 then for 510.csv fID = -1 (could not open file). Similarly when I deleted 50 files and it stopped at 560.csv the previous file 559.csv had fID = 511 and then for 560.csv fID = -1.

So what I gathered from of all of this was that the problem was not with any of the source files but that it was also stopping at the same location after fID = 511.

It's bizarre to me as the program works perfectly when I don't have as many files to process. It's also not a CPU/memory issue as I have been looking at the resource monitor when it halts and nothing is alarming.

Does anyone have any idea as to what is causing fopen to not find the next file when processing a large number of files?

Any help would be greatly appreciated.

Cheers.

[Stephen23]

You should always get and check the error message from fopen. Change that line to:

[fID,msg] = fopen(...);
assert(fID>=3,msg)

and tell us exactly what the message is.

PS: the intention of this is a bit clearer:

fullfile(file location,[file name,'.csv'])

than

strcat(fullfile( file location, file name),'.csv')

[HoboJoe]

Thanks for the feedback. I'll insert that and see what the exact error message is.

I won't be able to try it out until Monday so will report back then.

Cheers

Answer 1

[Walter Roberson]

If you were fclose()'ing the file properly, the file ID would not continuing increasing. So some code execution path is failing to fclose() the file after use.

[HoboJoe]

Interesting. I'll have a look into that. So that would mean that the program is opening and reading each file one by one but not closing them.

Can this create an issue though? i.e. is there a maximum number of files that can be open using fopen?

I'll see if closing each file after opening has any effect on the outcome.

Cheers

[Walter Roberson]

Yes, there is a maximum number of files that can be open using fopen. According to https://msdn.microsoft.com/en-us/library/6e3b887c.aspx the stdio (standard C I/O library) default limit is 512 files; it is possible to make an operating system call to increase that to 2048.

[HoboJoe]

Well you learn something new everyday! Seems to me that's definitely causing the issue! too many open files. Closing them after processing should solve the issue then.

Answer 2

[Steven Lord]

I suspect you're opening the files but not closing them. You can check this using:

listOfFileIDs = fopen('all')

This newsgroup thread is from a couple years ago, but I believe you might find it relevant.

[HoboJoe]

Ok great thanks for that. Will check it out when I get a chance.