Large Text File Import
2 views (last 30 days)
Show older comments
I have 4GB text file included 9 headerlines, 24 columns of raw number data (cca 9M rows). How can I import data to matlab, separate columns and save them as MAT easily and quickly? I tried to use textscan but it gave me only first 266 rows.
fid = fopen('2013067b.txt') Out = textscan(fid,'%f %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f','delimiter', '\t', 'HeaderLines',9); fclose(fid)
Thank You All!
0 Comments
Answers (2)
dpb
on 7 Feb 2014
c=reshape(textread('2013067b.txt'),'%f',-1,'delimiter','\t','headerlines',9),24,[])';
Sometimes the older tools are the better...
0 Comments
per isakson
on 7 Feb 2014
Edited: per isakson
on 7 Feb 2014
"[...]textscan but it gave me only first 266 rows" Without seeing the error message it's difficult to say why that happened. There was a message? I guess there is an anomaly in line 267
.
The second column is text, %s, - or ? I missed that. However, you write "24 columns of raw number data". The examples below assumes numerical data.
Possibly, dlmread is an alternative. Try
Mi = reshape( magic(12), [], 3 );
fid = fopen('cssm.txt','w');
for jj = 1 : 9
fprintf( fid, 'Header line %i\n', jj );
end
fprintf( fid, '%f,%f,%f\n', transpose(Mi) );
fclose( fid );
Mo = dlmread('cssm.txt',',',9,0);
all(Mi==Mo)
Or is it too slow?
.
Three more alternatives:
fid = fopen('cssm.txt','r');
for jj = 1 : 9
str = fgetl( fid );
end
num = fscanf( fid, '%f,%f,%f\n', [3,inf] );
fclose( fid );
Mf = transpose( num );
num = textread( 'cssm.txt','%f', -1, 'delimiter',',', 'headerlines',9 );
Mtr = transpose( reshape( num, 3, [] ) );
fid = fopen('cssm.txt','r');
num = textscan( fid,'%f', inf, 'delimiter',',', 'headerlines',9 );
Mts = transpose( reshape( num{:}, 3, [] ) );
fclose( fid );
all(Mf==Mi)
all(Mtr==Mi)
all(Mts==Mi)
0 Comments
See Also
Categories
Find more on Large Files and Big Data in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!