How to analyse the performance of Neural Network using Neural Network Toolbox? The performance and training state graphs are attached. Kindly Check.

I am using ORL face Database for face recognition. There are 40 subjects each with 10 images each. The size of each image is 112 * 92 pixels. For feature extraction and dimensionality reduction, I used 2D- PCA. The feature matrix is now 112*8.

Following is the code after feature extraction --

 % TrainFaces --- shape is 240 * 112 *8 (contains 240 images (8 images of 40 persons))
 % ValFaces --- shape is 80 * 112 *8 (contains 80 images (2 images of 40 persons))
 % TestFaces --- shape is 80 * 112 *8 (contains 80 images (2 images of 40 persons))

 % train_target -- shape is 40 * 240 (1st bit is 1 if it is 1st image and all rest zero)
 % val_target -- shape is 40 * 240
 % test_target -- shape is 40 * 240

 EigenFaces = [TrainFaces ; ValFaces ; TestFaces];
 target = [train_target val_target test_target];

 %%%%%%%%%%Now we can convert feature matrix into feature vector  %%%%%%%%%%%%%

 input = [];
    for i = 1:size(EigenFaces,1)
        input = [input reshape(EigenFaces(i,:,:),size(EigenFaces,2)*size(EigenFaces,3),1)]; 
    end
 %%%%%%%%Creating Network and Training Them %%%%%%%%%%%

 setdemorandstream(491218382);
 net = patternnet(25);
 net.divideFcn = 'divideind';
 [net.divideParam.trainInd, net.divideParam.valInd, net.divideParam.testInd] =   divideind(400,1:240,241:320,321:400);     
  net.trainFcn = 'trainrp';
  [net,tr] = train(net,input,target);

 %%%%%%%%%%%%%Testing the network %%%%%%%%%%%%%%%

 predict = 0;  % Number of correctly identified images out of 80
     for i =1:80
         [tar_val tar_ind] = max(test_target);
         [out_val out_ind] = max(net(reshape(TestFaces(i,:,:),size(TestFaces,2)*size(TestFaces,3),1)));

         if tar_ind == out_ind
             predict = predict +1;
         end
     end
     predict

The predict always shows zero.

Following is the snapshot of the result. Kindly help me analysis from the graph. Any help would be really grateful.

1. nntraintool

2. Performance

3. Training State

4. Error Histogram

[Mazhar Bukhari]

1. nntraintool This tool you know is used to train the network, it is pretty simple. 2. Performance There are four lines, Train, Validation, test and Best. In-fact, Best (dotted) line represents that other line should lie on or near this (dotted) lines, then we can be confirmed that training has been done successfully. If any of the 3 (Training, Validation and Testing) lines meet or pass near the best (dotted) line, it means convergence has been done, it this is not the case, retrain the network. 3. Training state It represents the current progress / status of the training at a specific time while training is in progress, in your case, 6 validation errors are mentioned, it means when simultaneously 6 validation check errors are produced then training will stop. Validation check error means dataset has some problems, may be some instances are not understandable by the training algorithm etc. means due to dataset problems, validation check errors can be generated. In your case, 6 validation checks are used.

Error histogram. It represents errors that approach to Mean Squared Error (MSE), in your case, it is good. Less than 10000 instances are used for training purpose, approx. 3000 are used to validation purpose and rest of these are used for testing purpose revealing 0.04284 error.

[Batool Abbas]

1.performamce

2.training set

3.histrogram

4. all

kindly explain me all this. please

[Batool Abbas]

@Greg Heath @Mazhar Bukhari kindly answer me plsss

Answer 1

[Greg Heath]

The success of a NN design often depends on a fortunate set of random initial weights AND a reasonable value for the number of hidden nodes, H.

MY approach:

 1. Begin with the default H = 10. 
 2. Explicitly initialize the RNG before the training loop
 3. Design Ntrials = 10 or more nets in a loop. The different trials will have different random initial weights, different random trn/val/tst data divisions and different answers.
 4. tr will contain various details of the training. Immediately after training the command 

    tr = tr

 will divulge most of the important training details.
 5. If none of the 10 designs are satisfactory, I introduce an outer loop over hidden node values 
   e.g., for h = 10:2:20, etc

 6. I have posted many, many examples of the double loop design procedure in both the NEWSGROUP and ANSWERS. For classification try mixtures of the following searchwords:

 greg patternnet Ntrials ind2vec tutorial

 7. In your case tr.stop will indicate that training stopped because the validation error reached a minimum. This is obvious from the plots. 

 8. I would just start again with Ntrials = 10 or more different designs, etc

 9. Use ind2vec to convert the true class indices 1:40 to the target matrix and vec2ind to convert the output to assigned class indices. Then comparing true and assigned will result in the error count.

Hope this helps.

Greg

[Sai Kumar Dwivedi]

Thank you so much for your reply. I will try out the methods suggested by you and will let you know if it doesn't work.

Answer 2

[Mazhar Bukhari]

1. nntraintool This tool you know is used to train the network, it is pretty simple. 2. Performance There are four lines, Train, Validation, test and Best. In-fact, Best (dotted) line represents that other line should lie on or near this (dotted) lines, then we can be confirmed that training has been done successfully. If any of the 3 (Training, Validation and Testing) lines meet or pass near the best (dotted) line, it means convergence has been done, it this is not the case, retrain the network. 3. Training state It represents the current progress / status of the training at a specific time while training is in progress, in your case, 6 validation errors are mentioned, it means when simultaneously 6 validation check errors are produced then training will stop. Validation check error means dataset has some problems, may be some instances are not understandable by the training algorithm etc. means due to dataset problems, validation check errors can be generated. In your case, 6 validation checks are used.

Error histogram. It represents errors that approach to Mean Squared Error (MSE)

[ghulam murtaza]

You have helped me alot again I have little bit confusion in training state what does Mu and gradient values shows which graph will be accurate. Secondly in error histogram for best graph what should be error range