CSV file reading and applying loop to plot graphs

30 views (last 30 days)
Hello
How can I do it for multiple csv file name of "WT_201119.csv" ,"WT_201120.csv" contain 52 main header with column. All headers are constant in csv files. Above csv read data correctly using readtable command but only for few files. where I have to do changes? similar to above one csv file Because I got error- Where I have to do changes always so that it can detect the total number of header and rows which we wan b "Error using matlab.io.internal.shared.TextInputs/set.DataLines (line 101) 'DataLines' must be specified as a positive integer or a 2-column matrix of positive integers."y
clc
clear all
d=dir('WT_201119.csv'); % "*" will detect the sequence of your file
opts = detectImportOptions(d(1).name,"Delimiter",","); % already know it's csv, avoid detecting it
T = {};
for i=1:numel(d)
fid = fopen(d(i).name,'r');
nhdr = 8 + cell2mat(textscan(fid,'Total Channels:%f',1,'HeaderLines',5,'Delimiter',','));
fid = fclose(fid);
if nhdr == 8 % skip no data in file cases if that occurs (dpb conjecture)
warning(['No data. Skipped file: ' d(i).name])
continue
end
opts.DataLines = [nhdr+2 inf];
opts.VariableNamesLine = nhdr+1;
T{i} = readtable(d(i).name,opts)
end
See more (1)
  5 Comments
Mathieu NOE
Mathieu NOE on 2 Jun 2021
hello
sorry I don't understand the meaning of your second sentence... what do you want to retrieve or not ? and finally what was the issue in first place because now I am bit lost...
Arshey Dhangekar
Arshey Dhangekar on 2 Jun 2021
"Error using matlab.io.internal.shared.TextInputs/set.DataLines (line 101) 'DataLines' must be specified as a positive integer or a 2-column matrix of positive integers."
getting error below the line
opts.DataLines = [nhdr+2 inf];

Sign in to comment.

Accepted Answer

Jeremy Hughes
Jeremy Hughes on 3 Jun 2021
Without an example file, it's impossible to say why the code doesn't work as expected. I'm just working backwards from the error message and what I know about the returns of textscan.
I see the example uploaded in a comment for a different answer, but that file works as expected with the code you included.
  5 Comments
LO
LO on 3 Jun 2021
Edited: LO on 3 Jun 2021
I agree, that's what I meant.
I think before writing any code about this, Arshey should have clear in mind where are the data files coming from, how are they formatted and standardize the format as much as possible.
If you scroll though the CSV files there are multiple columns containing several errors and the "tabulation" of the data is also incoherent (besides the different file format).
Arshey Dhangekar
Arshey Dhangekar on 15 Jun 2021
Edited: Arshey Dhangekar on 15 Jun 2021
Hello Jeremy,
I have two csv file WT and both have fixed have 52 header. I tried to use below command to read however header is as Var1,Var2..to Var52 instead of Store no,Date and Time. How can I fix the header problem with importing all the datas from those files?
table = readtable('WT_201120.csv');
table1 = readtable('WT_.csv');

Sign in to comment.

More Answers (2)

LO
LO on 2 Jun 2021
Edited: LO on 3 Jun 2021
first import one single CSV file and select coumns and rows of what you want usually to import.
Generate the code by clicking the 'generate code' button, in the import menu (right side of the screen, up)
Then use this code and put it where IMPORT CODE is written here below
[logfname, pathname] = uigetfile('*.csv','Pick a log file'); %select a csv file
cd(pathname); % move to that folder
file_list = dir ('*.csv') ; % create a list of csv files based on folder content
conversion = struct2cell(file_list);
conversion1 = transpose(conversion);
name_list = (conversion1(:,1));
% these two limits give you the option to select only a certain file range
% but you could actually remove them and simply set the k in the for loop
% from 1 to Nr of files
first= 1; % set first file to analyze
last= 10; % set last file to analyze
for k= first:last
filename = [pathname name_list{k}];
delimiter = ',';
startRow = 5;
endRow = 11831; % you can remove this or set it to a level that would fit all your data logs
formatSpec = '%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%[^\n\r]';
fileID = fopen(filename,'r','n','UTF-8');
fseek(fileID, 2, 'bof');
dataArray = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'TextType', 'string', 'EmptyValue', NaN, 'HeaderLines', startRow-1, 'ReturnOnError', false, 'EndOfLine', '\r\n');
fclose(fileID);
data_table = table(dataArray{1:end-1}, 'VariableNames', {'Name','x1','x2','x3','x4','x5','x6','x7','x8','x9','x10','x11','x12','x13','x14','x15','x16'});
data_table = data_table(22:end,:); % this crops away the header part of your file (somehow the file format is not properly detected by matlab and there is some problem with formatting
clearvars filename delimiter startRow endRow formatSpec fileID dataArray ans;
size_t = size(data_table);
plot_size = sqrt(size_t(2));
if plot_size > floor(plot_size)
plot_size=plot_size+1;
end
for i = 1:size_t(2)
subplot(round(plot_size),round(plot_size)-1,i)
% pre allocate var space
time_vector = zeros(numel(data_table(:,i)),1);
var_vector = zeros(numel(data_table(:,i)),1);
for j = 1:numel(data_table(:,i))
time = table2array(data_table(j,1));
time(time == ' ') = [];
time_vector(j) = str2double(time);
var = table2array(data_table(j,i));
var=strrep(var, ' ', '');
var_vector(j) = str2double(var);
end
plot(time_vector,var_vector)
title('title here')
xlabel('label')
ylabel('label')
end
end
  6 Comments
LO
LO on 2 Jun 2021
Edited: LO on 3 Jun 2021
I have updated the script, it works. However there is an issue with the text formatting of your data: due to the presence of whitespaces between characters the variables are converted into NaNs so you cannot see the data. I have tried to eliminate them using the function (var is one of the values in your data array)
strip(var,char(133)) or the other char codes you can see below but it does not remove them.
char(133) Next line
char(160) Nonbreaking space
char(8199) Figure space
char(8239) Narrow no-break space
the strip function could replace the line "var=strrep(var, ' ', '');", which is meant to have analogous results (although strrep does not affect special whitespace characters I suppose).
The problem may be in the formatting of the text and how MATLAB reads it. Do you have by chance the older file you posted ?
Apart from this, it should work.

Sign in to comment.


Jeremy Hughes
Jeremy Hughes on 3 Jun 2021
You should look at what nhdr is when you see the error. It's likely an empty value which makes nhdr+2 empty, and [nhdr+2 inf] just [inf].

Products


Release

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!