Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Neural Network Performance Evaluation Criteria
Replies: 3   Last Post: Mar 9, 2013 2:31 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Greg Heath

Posts: 5,954
Registered: 12/7/04
Re: Neural Network Performance Evaluation Criteria
Posted: Mar 7, 2013 7:18 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Louis " <olivier.louisez@gmail.com> wrote in message <kh5m8u$s2a$1@newscl01ah.mathworks.com>...
> Hai All,
>
> I have a question regarding performance evaluation criteria of a neural network. I >supposed to evaluate the network performance only through training examples


OH NO!

This has to be done VERY carefully because if you use the same data to train and evaluate, you will not have an unbiased estimate of how the net will generalize, i.e, perform on nontraining data, even if the data was obtained from the same source.
That is why an informed client will always insist that the model he/she is paying for
be evaluated on holdout nondesign data.

The basic problem is that with a I-H-O node topology and Ntrn pairs of I-dimensional inputs and O-dimensional output targets , you will have Ntrneq = Ntrn*O training equations to estimate Nw = (I+1)*H+(H+1)*O unknown weights. The estimation degrees of freedom is Ndof = Ntrneq - Nw (See Wikipedia).

The evaluation estimate MSEtrn = SSEtrn/Ntrneq is biased from using the same data to train and evaluate. To some extent the bias can be mitigated by using the degree-of-freedom-"a"djusted value MSEtrna = SSEtrn/Ndof. Obviously, the result becomes untenable as Ndof -> 0 because of the combination of too small Ntrn and/or too large H.

Real world data is usually contaminated by noise, measurement error, signal inteference
numerical roundoff and truncation error. Therefore, if the number of equations equals the
number of unknowns, the net will memorize a particular combination of signal and contamination and will not generalize well to other combinations.

A good approach is to have Ntrneq >> Nw and obtain a least squared error solution which tends to follow the signal and ignore the contamination. When Nw > Ntrneq, the
net is said to be OVERFIT because there are more unknown weights than constraint
equations to force a useful minimum error solution. However, if the signal dominates the contamination, an iterative solution will tend to pass through a useful solution on it's way to minimization via learning the contamination. Continuing beyond that point is called OVERTRAINING.

There are several ways to avoid overtraining an overfiit net. The most common are

% 1. Do not overfit
% a. Increase Ntrn
% and/or
% b. Reduce H
% 2. If overfit, do not overtrain
% a. Use validation subset stopping
% and/or
% b. Use regularization (help trainbr, help msereg) .

Although a validation set is used in the design to stop overtraining, if it is large enough, it's performance will be a much better estimate of generalization error than that obtained from training data.

I use the R^2 statistic to evaluate nets. See Wikipedia and my various posts in the
NEWSGROUP and ANSWERS. Search using Greg and one or two of my keywords
R2a, R2trna, Nw, Neq, Ntrneq, Ndof

Also see the comp.ai.neural-nets FAQ and posts re overfitting/training.

>and through testing the network. But, in the neural network toolbox, there is validation . > also. What is the advantages of the validation inspite of training the samples?

Preventing OVERTRAINING.

> Another remark is, I am using Matlab 2009b, instead i am also going to see if the validation criteria makes good result, but, i don't like to do randomization of the data because my training examples are almost time series. How could i used the data division in Matlab 2009b, since the net.dividefcn='dividetrain' does not support in my version. How can i use the data into train/validation/test without 'dividerand', which randomize data and not useful to me? . Only reason is that this randomization destroys my periodic series. Please, could you show sample code?

help/doc divideblock

help/doc divideind

ope this helps.

Greg



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.