Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
NEURAL NETWORK DESIGN: NONLINEAR REGRESSION EXAMPLE II

Subject: NEURAL NETWORK DESIGN: NONLINEAR REGRESSION EXAMPLE II

From: Greg Heath

Date: 27 Aug, 2013 01:17:08

Message: 1 of 1

This is a continuation of the previous post which could use resubstituted training data to estimate performance on nontraining data (generalization) because the number of training equations sufficiently exceeded the number of unknown weights.

In this post, only enough training data is used to adequately characterize the input/output transformation but not large enough for useful generalization performance estimation. The rules of thumb posted recently are used to guide the design. The results show that when the number of hidden nodes is large enough to cause OVERFITTING ( the estimation number of degrees of freedom is negative and useless for mitigating the bias of estimating generalization error by resubstituting training data), performance on nondesign holdout test data can be satisfactory (not usually the case).

close all, clear all, clc, plt = 0;
tic
[ x, t ] = simplefit_dataset;
[ I N ] = size(x) % [ 1 94 ]
[ O N ] = size(t) % [ 1 94 ]

plt=plt+1, figure(plt) % figure 1
plot( x, t, 'LineWidth', 2)
title( ' SIMPLEFIT DATASET ')
Nlocmax = 2.5 % 2.5 local maxima

xt = [ x; t ];
rangext = minmax(xt)
% rangext = 0 9.9763
% 0 10
% No need to standardize or normalize

Ntrn = ceil(5*Nlocmax) % 13
trnratio = Ntrn/N % 0.1383
valratio = 0 % 0
tstratio = 1-trnratio-valratio % 0.8617
[trnind valind tstind] = ...
                 divideint(N,trnratio,valratio,tstratio) ;
xtrn = x(trnind);
ttrn = t(trnind);
Ntst = length(tstind) % 81
xtst = x(tstind);
ttst = t(tstind);

plt=plt+1, figure(plt) % figure 2
hold on
plot( xtrn, ttrn, 'o', 'LineWidth', 2 )
plot( xtst, ttst, '.', 'LineWidth', 2 )
legend( 'TRAIN','TEST')
title( ' SIMPLEFIT DATA DIVISION ')

Ntrneq = prod(size(ttrn)) % 13
MSEtrn00 = mean(var(ttrn',1)) % 8.4395
MSEtrn00a = mean(var(ttrn')) % 9.1428
MSEtst00 = mean(var(ttst',1)) % 8.3214

% H >= 2*Nlocmax = 5
% Nw = (I+1)*H+(H+1)*O;
% Ntrneq >Nw ==> H <= Hub
Hub = -1+ceil( (Ntrneq-O) / (I+O+1)) % 3
Hmax = 2*Nlocmax +1 % 6
dH = 1
Hmin = 0

j=0
rng(0)
for h = Hmin:dH:Hmax
    j=j+1;
    h=h
    if h==0
        net = fitnet([]);
        Nw = (I+1)*O
    else
        net = fitnet(h);
        Nw = (I+1)*h+(h+1)*O
    end
    Ndof = Ntrneq-Nw
    net.divideFcn = 'divideind';
    net.divideParam.trainInd = trnind;
    net.divideParam.valind = [];
    net.divideParam.testInd = tstind;
    [ net tr y ] = train(net,x,t);
    
    plt = plt+1,figure(plt)
    hold on
    plot( xtrn, ttrn, 'o', 'LineWidth', 2 )
    plot( xtst, ttst, '.', 'LineWidth', 2 )
    plot( x, y, 'r--', 'LineWidth', 2 )
    legend( 'TRAIN', 'TEST', 'OUTPUT' )
    title( [ ' No. HIDDEN NODES = ', ...
                          num2str(h) ], 'LineWidth', 2 )
    
    stopcrit{j,1} = tr.stop;
    numepochs(j,1) = tr.num_epochs;
    bestepoch(j,1) = tr.best_epoch;
    MSEtrn(j,1) = tr.perf(tr.best_epoch+1);
    MSEtrna(j,1) = Ntrneq*MSEtrn(j,1)/Ndof;
    MSEtst(j,1) = tr.tperf(tr.best_epoch+1);
    
end

stopcrit = stopcrit

% stopcrit =
%
% 'Minimum gradient reached.'
% 'Minimum gradient reached.'
% 'Minimum gradient reached.'
% 'Minimum gradient reached.'
% 'Minimum gradient reached.'
% 'Minimum gradient reached.'
% 'Minimum gradient reached.'

H= (Hmin:dH:Hmax)';
R2trn = 1-MSEtrn/MSEtrn00;
R2trna = 1-MSEtrna/MSEtrn00a;
R2tst = 1-MSEtst/MSEtst00;
format short g
summary = [ H bestepoch R2trn R2trna R2tst]
toc

% summary =
% H bestepoch R2trn R2trna R2tst
% 0 2 0.52315 0.4798 0.54887
% 1 17 0.85138 0.80184 0.82564
% 2 47 0.91656 0.83313 0.85483
% 3 43 0.91656 0.66626 0.85483
% 4 501 0.99997 -Inf 0.99491
% 5 532 0.99997 1.0001 0.99491
% 6 24 1 1 0.99753
%
% Elapsed time is 16.2 seconds.

Hope this helps.

Greg

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us