Textscan function ignores final delimiter when token is empty

%str = 'a,b,c';  % works, yields 3 tokens
%str = 'a,,c';  % works, yields 3 tokens
str = 'a,b,';  % doesn't work, yields 2 tokens

line = textscan(str, '%q', 'Delimiter', ',', 'MultipleDelimsAsOne', 0);
disp(line{1})

    {'a'}
    {'b'}

textscan function seems to ignore the final delimiter if token is empty. 'a,b,' should be 3 tokens and it only generates 2

Answer 1

[Rik]

From what I can tell, this is intended functionality. It is unable to match the format specification to an empty array, so it is skipped.

If you want the trailing empty token, you may consider either the split() function, or regexp() with the split output.

Edit: here you have the examples I meant.

str{1,1} = 'a,b,c';
str{2,1} = 'a,,c';
str{3,1} = 'a,b,';
regexp(str,',','split') % three tokens each

ans = 3×1 cell array
    {1×3 cell}
    {1×3 cell}
    {1×3 cell}

split(str,',')

ans = 3×3 cell array
    {'a'}    {'b'     }    {'c'     }
    {'a'}    {0×0 char}    {'c'     }
    {'a'}    {'b'     }    {0×0 char}

% To show your solution works as well
cellfun(@(x)disp(textscan([x ','], '%q', 'Delimiter', ',', 'MultipleDelimsAsOne', 0)),str)

    {3×1 cell}

    {3×1 cell}

    {3×1 cell}

As you can see, each solution returns 3 tokens, but all in a slightly different format.

[Derek Wolfe]

Why would it read 'a,,c' as 3 tokens with an empty char in the middle, but not 'a,b,' as 3 tokens with an empty char at the end?

[Rik]

Because you told it not to merge delimiters. But the last one doesn't have a trailing token. You know there should be an empty trailing token, but that is not what this function is meant to do. Inserting extra (empty) tokens is already an extension.

I suspect the reason is to avoid unexpected results when used to scan a file with numbers.

But both alternatives I mentioned will give you a trailing empty token. Did you try it?

[Derek Wolfe]

I don't think it should work this way so I submitted a bug. Thanks for the ideas but it would be easier to workaround by adding an extra delimiter to the end of the string before scanning it

str = 'a,b,';
str = [str, ','];

line = textscan(str, '%q', 'Delimiter', ',', 'MultipleDelimsAsOne', 0);
disp(line{1})

    {'a'     }
    {'b'     }
    {0×0 char}

[Rik]

That works too.

Regarding your bug report: since the current behavior matches the description in the documentation, I don't see how this could be classified as a bug. You could call this an enhancement request.

Either way, your main problem is solved. Just be aware that textscan will try to read your format, not tokens. If you want tokens you need something like regexp.

Answer 2

[Walter Roberson]

filename = tempname() + ".txt";
str = {'12,453   ,  c'};
writelines(str, filename);

fid = fopen(filename, 'r');
datacell = textscan(fid, '%f%f', 1, 'Delimiter', ',', 'MultipleDelimsAsOne', 0);
rest = char(fread(fid, [1 inf], '*uchar'));
fclose(fid);

datacell

datacell = 1×2 cell array
    {[12]}    {[453]}

rest

rest = 
    '  c
     '

What does this tell us? Well, it tells us that after textscan finishes processing the format, it examines the input stream, consuming Whitespace and up to the first copy of the Delimiter, and then stops.

So in your example when processing 'a,b,' with %q it first processes the a as part of the %q format, and then eats the delimiter that is there, leaving 'b,' in the input stream. No count is specified so it tries processing again, reads the b with the %q format, eats the delimiter. No count is specified so it tries processing again. The input stream is empty, so it stops. Yes, this is the same thing that would happen for the case of 'a,b' with no final delimiter.

A potential alternative would have been if the delimiter was left in the buffer, and it was up to the processing to skip one leading delimiter. But in such a case the input ',b,c' would be treated the same as 'b,c' and it seems unlikely to me that you would want that to be the case.

Yes, textscan could have been designed to set an internal flag indicating that a delimiter had been seen before the current position, but I am not sure that would meet with expectations. For example if the input were 123,abc,<newline>456,def, and the format were %f%q then with the current behaviour that would generate {{123;456}, {'abc'; 'def'}} . If trailing delimiters should invoke the missing-contents behaviour (your interpretation) then that would imply that the 123 should be accepted by the first time %f is processed, then the abc should be accepted by the first time %q is processed, then the empty field at the end of the line should be accepted and turned into nan by the second time %f is processed, then the 456 should be accepted and turned into character vector by the second time %q is processed, then the def should trigger a format mismatch the third time %f is processed, stopping the scanning and leaving def, in the buffer. It might be potentially be a consistent way of processing, but is it what would be expected and useful ?