Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Neural Network Performance Evaluation Criteria
Replies: 3   Last Post: Mar 9, 2013 2:31 AM

 Messages: [ Previous | Next ]
 Greg Heath Posts: 6,387 Registered: 12/7/04
Re: Neural Network Performance Evaluation Criteria
Posted: Mar 7, 2013 7:18 AM

"Louis " <olivier.louisez@gmail.com> wrote in message <kh5m8u\$s2a\$1@newscl01ah.mathworks.com>...
> Hai All,
>
> I have a question regarding performance evaluation criteria of a neural network. I >supposed to evaluate the network performance only through training examples

OH NO!

This has to be done VERY carefully because if you use the same data to train and evaluate, you will not have an unbiased estimate of how the net will generalize, i.e, perform on nontraining data, even if the data was obtained from the same source.
That is why an informed client will always insist that the model he/she is paying for
be evaluated on holdout nondesign data.

The basic problem is that with a I-H-O node topology and Ntrn pairs of I-dimensional inputs and O-dimensional output targets , you will have Ntrneq = Ntrn*O training equations to estimate Nw = (I+1)*H+(H+1)*O unknown weights. The estimation degrees of freedom is Ndof = Ntrneq - Nw (See Wikipedia).

The evaluation estimate MSEtrn = SSEtrn/Ntrneq is biased from using the same data to train and evaluate. To some extent the bias can be mitigated by using the degree-of-freedom-"a"djusted value MSEtrna = SSEtrn/Ndof. Obviously, the result becomes untenable as Ndof -> 0 because of the combination of too small Ntrn and/or too large H.

Real world data is usually contaminated by noise, measurement error, signal inteference
numerical roundoff and truncation error. Therefore, if the number of equations equals the
number of unknowns, the net will memorize a particular combination of signal and contamination and will not generalize well to other combinations.

A good approach is to have Ntrneq >> Nw and obtain a least squared error solution which tends to follow the signal and ignore the contamination. When Nw > Ntrneq, the
net is said to be OVERFIT because there are more unknown weights than constraint
equations to force a useful minimum error solution. However, if the signal dominates the contamination, an iterative solution will tend to pass through a useful solution on it's way to minimization via learning the contamination. Continuing beyond that point is called OVERTRAINING.

There are several ways to avoid overtraining an overfiit net. The most common are

% 1. Do not overfit
% a. Increase Ntrn
% and/or
% b. Reduce H
% 2. If overfit, do not overtrain
% a. Use validation subset stopping
% and/or
% b. Use regularization (help trainbr, help msereg) .

Although a validation set is used in the design to stop overtraining, if it is large enough, it's performance will be a much better estimate of generalization error than that obtained from training data.

I use the R^2 statistic to evaluate nets. See Wikipedia and my various posts in the
NEWSGROUP and ANSWERS. Search using Greg and one or two of my keywords
R2a, R2trna, Nw, Neq, Ntrneq, Ndof

Also see the comp.ai.neural-nets FAQ and posts re overfitting/training.

>and through testing the network. But, in the neural network toolbox, there is validation . > also. What is the advantages of the validation inspite of training the samples?

Preventing OVERTRAINING.

> Another remark is, I am using Matlab 2009b, instead i am also going to see if the validation criteria makes good result, but, i don't like to do randomization of the data because my training examples are almost time series. How could i used the data division in Matlab 2009b, since the net.dividefcn='dividetrain' does not support in my version. How can i use the data into train/validation/test without 'dividerand', which randomize data and not useful to me? . Only reason is that this randomization destroys my periodic series. Please, could you show sample code?

help/doc divideblock

help/doc divideind

ope this helps.

Greg

Date Subject Author
3/7/13 Greg Heath
3/9/13 Greg Heath