How to inclusively extract rows of a large cell array between cells given start and end patterns?
Show older comments
Hello Folks,
I am searching for the most efficent method to parse a large text file (typically 2-4 GB) for ocurrences of a message. I have to search ~100 large files for dozens of messages so efficiency will be quite significant. I have attached a sample_input.txt with two occurrences of a message specified in the considerations below.
Considerations:
1) start of the message is: 'Hello_Message.pdf'
2) end of the message is: '&&&'
3) store all lines of each occurence of the message to an array within a structure
5) all messages have a header pattern '.*\.[a-zA-Z]{3}\n\r' and end with pattern '&&&\n\r'
4) hopeful to avoid for loops by filtering using a function for example extractBetween, Contains, regexpPattern, or other function(s)
The code below does not work but hopefully it provides an idea of what I was thinking...
clear
close all
clc
Input_fid = fopen(sample_input.txt);
ftext = textscan(Input_fid,'%s','Delimeter','\n\r');
fclose(Input_fid)
% I want to inclusively capture the start of the message 'Hello_Message.pdf' and the end
% of the message '&&&' along with all rows between the start and end of each ocurrence
% of the message
for check = 1:height(ftext{1})
HelloMsgs.Occurrences(check) = extractBetween(ftext{1},regexpPattern('Hello_Message.pdf.*\n\r'),regexpPattern('&&&\n\r'));
end
Desired Output:
HelloMsgs.Occurrences(1) <--- cell array of all lines of first occurrence of the Hello_Message in its
own row cell
HelloMsgs.Occurrences(2) <--- cell array of all lines of second occurrence of the Hello_Message in its
own row cell
HelloMsgs.Occurrences(3) <--- cell array of all lines of third occurrence of the Hello_Message in its
own row cell
Thank you in advance for your time. I am new to posting a coding question in a forum so hopefully I explained
the problem well enough.
4 Comments
There are anomalies in the file that keep this approach from working correctly.
My attempt —
type('sample_input.txt')
fidi = fopen('sample_input.txt','rt')
k = 1;
while ~feof(fidi)
Line{k,:} = fgetl(fidi);
k = k+1;
end
fclose(fidi);
k
Line
for k1 = 1:k-1
if ~isempty(Line{k1,:})
if strmatch(Line{k1,:},'Hello_Message.pdf')
% Start(k1) = 1
sprintf('Start = %2d',k1)
end
if strmatch(Line{k1}, '&&&')
% End(k1) = 1;
sprintf('End = %2d',k1)
end
end
end
.
Star Strider
on 18 Oct 2023
You specifically stated:
‘start of the message is: 'Hello_Message.pdf'’
so that is all I considered. If you want to get all of them, there are ways to do that, fopr example the extractBefore function, and then compare only the part up to the end of the file prefix. I changed it in my posted Answer.
Jude
on 18 Oct 2023
Star Strider
on 18 Oct 2023
Thank you.
I substituted extractBetween for extractBefore since that gives the appropriate result in my ‘Extract’ cell array.
Accepted Answer
Star Strider
on 18 Oct 2023
Edited: Star Strider
on 18 Oct 2023
type('sample_input.txt')
fidi = fopen('sample_input.txt','rt');
k = 1;
while ~feof(fidi)
Line{k,:} = fgetl(fidi);
k = k+1;
end
fclose(fidi);
k
% Line
for k1 = 1:k-1
if ~isempty(Line{k1,:})
Lc = strfind(extractBetween(Line{k1,:},'_','.'),'Message');
if ~isempty(Lc)
Start(k1) = 1;
% sprintf('Start = %2d',k1)
end
if strfind(Line{k1}, '&&&')
End(k1) = 1;
% sprintf('End = %2d',k1)
end
end
end
StartIdx = find(Start)
EndIdx = find(End)
for k = 1:numel(StartIdx)
Extract{k,:} = Line(StartIdx(k):EndIdx(k));
end
Extract{1}
Extract{end}
EDIT — (18 Oct 2023 at 03:42)
I am a bit lost with respect to ‘start key’ and ‘stop key’. My code defines ‘StartIdx’ and ‘StopIdx’ as the indices that define the ‘Message’ and ‘&&&’ entries. The ‘Extract’ cell arrays are those lines and all the lines between them.
My initial approach was to use the fileread function and then do ‘logical indexing’, however that failed so the loop was the only other available option.
My code here is the same code I posted as a Comment, changed to test for all the ‘Message’ lines and not only ‘Hello_Message.pdf’ that was initially specified.
.
6 Comments
Jude
on 18 Oct 2023
I just now ran this and it seems to do what you want.
What specifically would you want to change?
clear
clc
fidi = fopen('sample_input.txt','rt');
k=1;
while ~feof(fidi)
Line{k,1} = fgetl(fidi);
k = k+1;
end
fclose(fidi);
for k1 = 1:k-1
if strfind(Line{k1,1}, 'Hello_Message.pdf')
Start(k1)=1;
StartIdx = find(Start);
for k2 = k1+1:k-1
if strfind(Line{k2,1}, '&&&')
End(k2) = 1;
EndIdx = find(End);
break
end
end
end
end
for k = 1:numel(StartIdx)
Extract{k,1} = Line(StartIdx(k):EndIdx(k)); % contains the solution I am looking for
end
Filtered_Msgs_StartEnd = {StartIdx,EndIdx}
Cnt_Filtered_Msgs = numel(Filtered_Msgs_StartEnd)
Extract{:}
.
Jude
on 19 Oct 2023
Without altering my previous code significantly, I added an extra for loop to extract the next ‘EndIdx’ value greater than the preceeding ‘StartIdx’ value, and then saving those values (initially assigned to ‘NextEnd’) to ‘EndIdx’ afterwards. That produced two sets of consecutive ‘’StartIdx’ and ‘EndIdx’ values for each section.
type('sample_input.txt')
fidi = fopen('sample_input.txt','rt');
k = 1;
while ~feof(fidi)
Line{k,:} = fgetl(fidi);
k = k+1;
end
fclose(fidi);
k
% Line
for k1 = 1:k-1
if ~isempty(Line{k1,:})
Lc = strfind(Line{k1,:}, 'Hello_Message.pdf');
% Lc = strfind(extractBetween(Line{k1,:},'_','.'),'Message');
if ~isempty(Lc)
Start(k1) = 1;
% sprintf('Start = %2d',k1)
end
if strfind(Line{k1}, '&&&')
End(k1) = 1;
% sprintf('End = %2d',k1)
end
end
end
StartIdx = find(Start);
EndIdx = find(End);
for k = 1:numel(StartIdx)
NextEnd(k) = EndIdx(find(EndIdx > StartIdx(k), 1));
end
StartIdx
EndIdx = NextEnd
for k = 1:numel(StartIdx)
Extract{k,:} = Line(StartIdx(k):EndIdx(k));
end
Extract{1}
Extract{end}
This approach (adding an additional loop) is likely the most efficient way to choose the correct ‘EndIdx’ for each ‘StartIdx’.
.
Jude
on 20 Oct 2023
Star Strider
on 20 Oct 2023
As always, my pleasure!
More Answers (1)
Why do you want to avoid loops? Reading the file completely to apply vectorized methods requires 8 GB of contiguous free RAM for a 4 GB file (16 bit per char). I'd choose such an approach only on computers with >= 32 GB RAM, while a loop method is less demanding concering the RAM. In addition a filtering during the reading avoid to keep the complete text in the RAM.
S = ParseFile("sample_input.txt");
S{1}
function S = ParseFile(File)
startKey = "Hello_Message.pdf";
stopKey = "&&&";
fid = fopen(File, 'r');
assert(fid > 0, "Cannot open file: %s", File);
bS = 1000; % Pre-allocate output in blocks
nS = bS;
iS = 0;
S = cell(1, nS);
buffer = cell(20, 1); % Grows iteratively at first
ibuffer = 0;
doGrab = false;
while ~feof(fid)
Line = fgetl(fid);
if startsWith(Line, startKey)
buffer(:) = {[]}; % Clear the buffer
ibuffer = 0;
doGrab = true; % Start grabbing in next line
elseif startsWith(Line, stopKey)
doGrab = false; % Stop grabbing
iS = iS + 1; % Expand output S in blocks on demand
if iS > nS
nS = nS + bS;
S{nS} = [];
end
S{iS} = buffer(1:ibuffer); % Store the buffer
elseif doGrab
ibuffer = ibuffer + 1;
buffer{ibuffer} = Line;
end
end
fclose(fid);
if doGrab % Store last buffer, if stopKey is missing?!?
iS = iS + 1;
S{iS} = Line;
end
S = S(1:iS); % Crop pre-allocated output cells
end
Categories
Find more on Language Support in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
