Load multiple text files one after another

11 views (last 30 days)
Hello,
I have 30 text files with the format "Data1.x_pre" and "Data1.x_post" with x being numbered from 1-30. I want to load in and run through one MATLAB script. Currently, I am loading all of 30 text files separately which is quite tediuos. What I basically want to do is load the first text file ("Data1.1_pre"), run it through the script and collecting the output in a new matrix "Alldata". Then, I'd like to close the first text file and continue with the second file ("Data1.1_post") and insert the output into matrix "Alldata". After that: Closing "Data1.1_post", loading next text file ("Data1.2_pre") and so on.
My section to load and read these text files:
textFilename = ['Data1.1pre.txt'];
fid = fopen(textFilename);
block = 1;
newLine = fgets(fid);
while ~feof(fid);
EEGData{1,q}(1,block) = textscan(fid, '%f %f %f %f', 'CollectOutput', true);
fgets(fid);
block = block + 1;
end
Is there a way to solve these two problems (reading all text files and collecting all the outputs into one common matrix) with only little changes to my current code?
Thank you in advance!

Accepted Answer

per isakson
per isakson on 18 Sep 2014
Edited: per isakson on 24 Sep 2014
&nbsp
After 17 comments it's time I add some working code. I still don't understand the intended stucture of the cell array, EEGData. The following code is my solution to OP's underlying problem as I understand it. My steps:
  • download of Data1.01_pre.txt and creation of another five files by copy&rename.
  • move of the six files to a separate folder, my_eeg_data. I think it helps to store experimental data in dedicated folders. However, the function, cssm, doesn't depend on a separate data folder.
  • creation of the function, cssm (attached). I find it easier to develop and use functions compared to scripts. I understand that OP wish to use the "filename" as key to reference the data. (The names of the files are not valid Matlab variable names.)
  • demo of the function, cssm
Demo:
>> eeg_data_pre = cssm( 'h:\m\cssm\my_eeg_data', 'Data*pre*.txt' )
>> eeg_data_post = cssm( 'h:\m\cssm\my_eeg_data', 'Data*post*.txt' )
>> eeg_data = cat( 1, eeg_data_pre, eeg_data_post );
>> keys(eeg_data)
ans =
Columns 1 through 3
'Data1.01_post.txt' 'Data1.01_pre.txt' 'Data1.02_post.txt'
Columns 4 through 6
'Data1.02_pre.txt' 'Data1.03_post.txt' 'Data1.03_pre.txt'
>> num = eeg_data( 'Data1.02_post.txt' );
>> whos num
Name Size Bytes Class Attributes
num 12x20 1920 double
eeg_data is an instance of containers.Map
&nbsp
cssm (I use body text to comment the code.)
The function comprises to parts: creation of a "list" of file names and a loop over all file names to read the data
function eeg_data = cssm( folder, glob )
dir is a robust way to retrieve the names of the files. It avoids the problem of reconstructing the names.
sad = dir( fullfile( folder, glob ) );
Possibly the "list" requires some manipulations. Here I sort it with respect to the number, which is between the dot and the underscore. However, with the leading zero as in this case it is not needed. And with the solution in this function it is not needed anyhow.
cac = regexp( {sad.name}, '(?<=\.)\d{1,2}(?=_)', 'match' );
num = str2double( [cac{:}] );
[~,ixs] = sort( num );
sad = sad( ixs );
len = length( sad );
The number of columns could have been retrieved from the text file.
nCol = 20;
Here I use containers.Map because it makes it possible to use the names of the files as key values. Had the names of the files been valid Matlab names I would have used a struct.
eeg_data = containers.Map( 'KeyType', 'char', 'ValueType', 'any' );
for jj = 1 : len
filespec = fullfile( folder, sad(jj).name );
[fid,msg] = fopen( filespec );
This test might not be justified since the names are created by dir
assert( not(fid==-1) ...
, 'MY:eeg_data:CannotOpenFile'...
, 'Failed to open "%s": "%s"' ...
, filespec, msg )
I read the data to a temporary variable. I use repmat to create the format string. It saves me from counting to twenty. The defaults takes care of the list delimiter.
buf = textscan( fid, repmat('%f', [1,nCol] )...
, 'CollectOutput' , true ...
, 'Headerlines' , 1 );
eeg_data( sad(jj).name ) = buf{:};
end
end
&nbsp
For me this was a worthwhile exercise with containers.Map. This is the first time I concatenate maps. Is that documented or does it go without saying?

More Answers (2)

Pinga
Pinga on 19 Sep 2014
Thank you for the link. Unfortunately, this was not quite the solution I was looking for, since the script contains "data-relative" functions.
Is there no function at all which I could put at the end of the script telling MATLAB to clear the workspace and to start again from the beginning with the next .txt-file? I don't want to copy the script 30 times below the existing.
Any help would be appreciated.
  6 Comments
Pinga
Pinga on 22 Sep 2014
Thank you for your help. The exist error message has disappeared now. And no, the data files aren't in a special folder. The script and the data files are all in the same folder.
per isakson
per isakson on 23 Sep 2014
I've added a working function to my answer.

Sign in to comment.


Guillaume
Guillaume on 21 Sep 2014
To format your filenames:
for filenumber = 1:30
for postpre = {'post', 'pre'}
filename = sprintf('Data1.%d_%s.txt', filenumber, postpre{1});
if exist(filename, 'file')
%open file and process
else
%report error
end
end
end
if the for postpre = ... looks too weird for you, you can replace it with
postpre = {'post', 'pre'};
for postpreindex = 1:2
filename = sprintf('Data1.%d_%s.txt', filenumber, postpre{postpreindex});
  14 Comments
per isakson
per isakson on 24 Sep 2014
"which says "Invalid file identifier" &nbsp I guess the problem is the format string in
filename = sprintf('Data1.%d_%s.txt', filenumber, postpre{1});
The name of the file, which you uploaded is
Data1.01_pre.txt
To get the leading zero, i.e .01_ not .1_, the format string should be
'Data1.%02d_%s.txt'
Pinga
Pinga on 26 Sep 2014
Thank you both for your help - you've been very patient and helpful! There still were some issues and I've been debugging the last two days to get rid of them but it seems now that this part of the script works.
Thank you again and have a great day!

Sign in to comment.

Categories

Find more on Variables in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!