How do I find initial point of an exponential graph given a set of data points?

I have a noisy signal imported from a 24k x 1 excel sheet that was passed through a guassian filter to smooth out the curves as seen below.

What i need to find is the initial point in which the graph starts to turn exponentially as seen below, denoted as the black x.

I tried findchangepoints however, it gives me the red x as shown in the pictures as it only marks the points with the highest change.

ChangePts = findchangepts(graph,'Statistic','linear','MaxNumChanges',2);

Anybody knows any functions or methods that i should look into to solve this?

Edit: Just added the data.mat file. Kept forgetting to do so, sorry guys for the trouble.

[Rik]

If the first part is guaranteed to be linear, you could fit a linear curve to the first N points and see for what time the fit is not good enough anymore, or the right-most value deviates from the line too much.

[Wangping Chen]

The first "linear" part isn't exactly linear, but i get what you mean. I googled "linear curve fitting" and found out that matlab has a curve fitting app, I will try using it. Ty for the information!

[Rik]

The polyfit function can also fit a linear function, and doesn't require a toolbox, so you could also try that first.

[Sam Chak]

Hi @Wangping Chen

Perhap you can try computing the derivative of the sigmoidal curve. The derivative should look like an inverted bell-shaped curve. Then you can find the interval that corresponds to the (two standard deviations), which I believe it is very near to the positions that you marked with black x.

[Wangping Chen]

Hi @Sam Chak

I was thinking about that, but I will need to know the actual best fit equation for the graph. Since this is graph plotted with a set of arrays, I don't know how to find it.

Is there any functions or methods that allows us to do just that? I know excel has a function that does that with polynomials, exponentials and linear graph. Is there anything similar to that in Matlab?

[Rik]

There are many curve fitting tools in Matlab, many of them are in the Curve Fitting Toolbox.

Do you know the shape of the formula you want to fit? Is it a normal sigmoid? (so s=1/(1+exp(-x)) but with weights)

[Wangping Chen]

Alright will give it try. Ty for pointing me at the right direction!

[Sam Chak]

Hi @Wangping Chen

You can definitely use the curve fitting toolbox as advised by @Rik.

Because I don't have the dataset, I extract some data points from your image. From the data points, I use the Curve Fitting Toolbox to enter a sigmoidal function, which is a hyperbolic tangent function, in the Custom Equation field, with a is the amplitude, b is the breadth (width), c is the center, and d is the displacement (offset).

[Wangping Chen]

Hi @Sam Chak,

I am really sorry for misleading you, but the full actual graph looks like this:

Thank you for your help again as you gave me another idea on how to find the points that I want by cropping parts of the graph of interest and finding the best fit equation for that particular section. However, is such line of thinking viable in terms of maintaining the accuracy of quantifying the data i was given? Or as long as the R^2 of the cropped graph is close to 1, it is still considered accurate enough?

That said here is my progress, I used smoothing spline and got this:

Which i went on to differetiate and got this (where the red X denotes the max and min point of the graph) which is clearly a wrong interpretation of the graph:

In which I plotted back to the frequency time graph and it seems to be completely out of place (denoted as the black circles):

I am not very well verse in the math of the derivatives of a smoothing spline and i am in the process of learning more about it. But given the information i have right now, is it possible to filter my differentiated frequency time graph through another guassian filter and plot my points from there? Or is there anything I am missing?

[Rik]

It will make matters easier if you post your data in a mat file. Feel free to attach it to your question so it doesn't get burried in the comments.

[Wangping Chen]

Sorry, just added the data file. Still new to matlab and this forum, thank you so far tho!

Answer 1

[Mathieu NOE]

hello

sorry, i'm coming late in the show

this is my result and the code below , using the second derivative, smoothing and threshold crossing detection

hope it helps

clc
clearvars

S = load('data.mat');
x = S.CropTime_1;
dx = mean(diff(x));

y = S.Crop_yggaus_D1_3;
dyg = gradient(y,dx);
ddyg = gradient(dyg,dx);
ddygs = smoothdata(ddyg,'gaussian',200);

% [pks,locs] = findpeaks(ddygs,'MinPeakHeight',max(ddygs)/2,'NPeaks',2);
threshold = max(ddygs)/20;
[t0_pos1,s0_pos1,t0_neg1,s0_neg1]= crossing_V7(ddygs,x,threshold,'linear'); % positive (pos) and negative (neg) slope crossing points 

x1 = t0_pos1(1);
x2 = t0_neg1(end);
y1 = interp1(x,y,x1);
y2 = interp1(x,y,x2);


figure(1)
subplot(211),plot(x,y,x1,y1,'*r',x2,y2,'+c','linewidth',2,'markersize',12);
subplot(212),plot(x,ddygs,t0_pos1,s0_pos1,'*r',t0_neg1,s0_neg1,'+c','linewidth',2,'markersize',12);grid on
legend('second derivative','positive slope crossing points','negative slope crossing points');



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [t0_pos,s0_pos,t0_neg,s0_neg] = crossing_V7(S,t,level,imeth)

% [ind,t0,s0,t0close,s0close] = crossing_V6(S,t,level,imeth,slope_sign) % older format

% CROSSING find the crossings of a given level of a signal
%   ind = CROSSING(S) returns an index vector ind, the signal
%   S crosses zero at ind or at between ind and ind+1
%   [ind,t0] = CROSSING(S,t) additionally returns a time
%   vector t0 of the zero crossings of the signal S. The crossing
%   times are linearly interpolated between the given times t
%   [ind,t0] = CROSSING(S,t,level) returns the crossings of the
%   given level instead of the zero crossings
%   ind = CROSSING(S,[],level) as above but without time interpolation
%   [ind,t0] = CROSSING(S,t,level,par) allows additional parameters
%   par = {'none'|'linear'}.
%	With interpolation turned off (par = 'none') this function always
%	returns the value left of the zero (the data point thats nearest
%   to the zero AND smaller than the zero crossing).
%


% check the number of input arguments
error(nargchk(1,4,nargin));

% check the time vector input for consistency
if nargin < 2 | isempty(t)
	% if no time vector is given, use the index vector as time
    t = 1:length(S);
elseif length(t) ~= length(S)
	% if S and t are not of the same length, throw an error
    error('t and S must be of identical length!');    
end

% check the level input
if nargin < 3
	% set standard value 0, if level is not given
    level = 0;
end

% check interpolation method input
if nargin < 4
    imeth = 'linear';
end


% make row vectors
t = t(:)';
S = S(:)';

% always search for zeros. So if we want the crossing of 
% any other threshold value "level", we subtract it from
% the values and search for zeros.
S   = S - level;

% first look for exact zeros
ind0 = find( S == 0 ); 

% then look for zero crossings between data points
S1 = S(1:end-1) .* S(2:end);
ind1 = find( S1 < 0 );

% bring exact zeros and "in-between" zeros together 
ind = sort([ind0 ind1]);

% and pick the associated time values
t0 = t(ind); 
s0 = S(ind);

if ~isempty(ind)

    if strcmp(imeth,'linear')
        % linear interpolation of crossing
        for ii=1:length(t0)
            %if abs(S(ind(ii))) >= eps(S(ind(ii)))    % MATLAB V7 et +
            if abs(S(ind(ii))) >= eps*abs(S(ind(ii)))    % MATLAB V6 et -    EPS * ABS(X)

                % interpolate only when data point is not already zero
                NUM = (t(ind(ii)+1) - t(ind(ii)));
                DEN = (S(ind(ii)+1) - S(ind(ii)));
                slope =  NUM / DEN;
                slope_sign(ii) = sign(slope);
                t0(ii) = t0(ii) - S(ind(ii)) * slope;
                s0(ii) = level;
            end
        end
    end

    % extract the positive slope crossing points 
    ind_pos = find(sign(slope_sign)>0);
    t0_pos = t0(ind_pos);
    s0_pos = s0(ind_pos);

    % extract the negative slope crossing points 
    ind_neg = find(sign(slope_sign)<0);
    t0_neg = t0(ind_neg);
    s0_neg = s0(ind_neg);

else

    % empty output
    ind_pos = [];
    t0_pos = [];
    s0_pos = [];

    % extract the negative slope crossing points 
    ind_neg = [];
    t0_neg = [];
    s0_neg = [];
    
end

end

[Wangping Chen]

Sorry for the late reply and thank you for your guidance. My end goal is getting the points at here (denoted in black):

As we seen here, we got the 1st and last point of the given graph.

However, shouldn't the crossing_V7 also account for the black x?

Since the graph above is suppose find the points at point time vector of the zero crossings of the signal, why isn't the black x being taken account?

Anyways to find those 2 points? I tried interpolation and polyxpoly, but the interpolation returns an empty array while polyxpoly does not seem to work. Any ideas?

[Mathieu NOE]

hello

I added the missing points

the trick is to use crossing_V7 once with a positive threshold and a second time with a negative threshold

clc
clearvars

S = load('data.mat');
x = S.CropTime_1;
dx = mean(diff(x));

y = S.Crop_yggaus_D1_3;
dyg = gradient(y,dx);
ddyg = gradient(dyg,dx);
ddygs = smoothdata(ddyg,'gaussian',200);

threshold1 = max(ddygs)/20;
[t0_pos1,s0_pos1,t0_neg1,s0_neg1]= crossing_V7(ddygs,x,threshold1,'linear'); % positive (pos) and negative (neg) slope crossing points 

x1 = t0_pos1(1);
x2 = t0_neg1(end);
y1 = interp1(x,y,x1);
y2 = interp1(x,y,x2);

threshold2 = min(ddygs)/20;
[t0_pos2,s0_pos2,t0_neg2,s0_neg2]= crossing_V7(ddygs,x,threshold2,'linear'); % positive (pos) and negative (neg) slope crossing points 
x3 = t0_pos2(1);
x4 = t0_neg2(end);
y3 = interp1(x,y,x3);
y4 = interp1(x,y,x4);

figure(1)
subplot(211),plot(x,y,x1,y1,'*r',x2,y2,'+c',x3,y3,'*g',x4,y4,'+k','linewidth',2,'markersize',12);
subplot(212),plot(x,ddygs,t0_pos1,s0_pos1,'*r',t0_neg1,s0_neg1,'+c',t0_pos2,s0_pos2,'*g',t0_neg2,s0_neg2,'+k','linewidth',2,'markersize',12);grid on
legend('second derivative','pos thr pos slope crossing points','pos thr neg slope crossing points','neg thr pos slope crossing points','neg thr neg slope crossing points');



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [t0_pos,s0_pos,t0_neg,s0_neg] = crossing_V7(S,t,level,imeth)

% [ind,t0,s0,t0close,s0close] = crossing_V6(S,t,level,imeth,slope_sign) % older format

% CROSSING find the crossings of a given level of a signal
%   ind = CROSSING(S) returns an index vector ind, the signal
%   S crosses zero at ind or at between ind and ind+1
%   [ind,t0] = CROSSING(S,t) additionally returns a time
%   vector t0 of the zero crossings of the signal S. The crossing
%   times are linearly interpolated between the given times t
%   [ind,t0] = CROSSING(S,t,level) returns the crossings of the
%   given level instead of the zero crossings
%   ind = CROSSING(S,[],level) as above but without time interpolation
%   [ind,t0] = CROSSING(S,t,level,par) allows additional parameters
%   par = {'none'|'linear'}.
%	With interpolation turned off (par = 'none') this function always
%	returns the value left of the zero (the data point thats nearest
%   to the zero AND smaller than the zero crossing).
%


% check the number of input arguments
error(nargchk(1,4,nargin));

% check the time vector input for consistency
if nargin < 2 | isempty(t)
	% if no time vector is given, use the index vector as time
    t = 1:length(S);
elseif length(t) ~= length(S)
	% if S and t are not of the same length, throw an error
    error('t and S must be of identical length!');    
end

% check the level input
if nargin < 3
	% set standard value 0, if level is not given
    level = 0;
end

% check interpolation method input
if nargin < 4
    imeth = 'linear';
end


% make row vectors
t = t(:)';
S = S(:)';

% always search for zeros. So if we want the crossing of 
% any other threshold value "level", we subtract it from
% the values and search for zeros.
S   = S - level;

% first look for exact zeros
ind0 = find( S == 0 ); 

% then look for zero crossings between data points
S1 = S(1:end-1) .* S(2:end);
ind1 = find( S1 < 0 );

% bring exact zeros and "in-between" zeros together 
ind = sort([ind0 ind1]);

% and pick the associated time values
t0 = t(ind); 
s0 = S(ind);

if ~isempty(ind)

    if strcmp(imeth,'linear')
        % linear interpolation of crossing
        for ii=1:length(t0)
            %if abs(S(ind(ii))) >= eps(S(ind(ii)))    % MATLAB V7 et +
            if abs(S(ind(ii))) >= eps*abs(S(ind(ii)))    % MATLAB V6 et -    EPS * ABS(X)

                % interpolate only when data point is not already zero
                NUM = (t(ind(ii)+1) - t(ind(ii)));
                DEN = (S(ind(ii)+1) - S(ind(ii)));
                slope =  NUM / DEN;
                slope_sign(ii) = sign(slope);
                t0(ii) = t0(ii) - S(ind(ii)) * slope;
                s0(ii) = level;
            end
        end
    end

    % extract the positive slope crossing points 
    ind_pos = find(sign(slope_sign)>0);
    t0_pos = t0(ind_pos);
    s0_pos = s0(ind_pos);

    % extract the negative slope crossing points 
    ind_neg = find(sign(slope_sign)<0);
    t0_neg = t0(ind_neg);
    s0_neg = s0(ind_neg);

else

    % empty output
    ind_pos = [];
    t0_pos = [];
    s0_pos = [];

    % extract the negative slope crossing points 
    ind_neg = [];
    t0_neg = [];
    s0_neg = [];
    
end

end

[Wangping Chen]

Thank you very much for helping me along! Still in the process of reading and understanding the codes for crossing_v7, but i have a better picture of the codes now!

[Mathieu NOE]

My pleasure !