Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
NCTM or The Math Forum.



Re: Neural Network Performance Evaluation Criteria
Posted:
Mar 7, 2013 7:18 AM


"Louis " <olivier.louisez@gmail.com> wrote in message <kh5m8u$s2a$1@newscl01ah.mathworks.com>... > Hai All, > > I have a question regarding performance evaluation criteria of a neural network. I >supposed to evaluate the network performance only through training examples
OH NO!
This has to be done VERY carefully because if you use the same data to train and evaluate, you will not have an unbiased estimate of how the net will generalize, i.e, perform on nontraining data, even if the data was obtained from the same source. That is why an informed client will always insist that the model he/she is paying for be evaluated on holdout nondesign data.
The basic problem is that with a IHO node topology and Ntrn pairs of Idimensional inputs and Odimensional output targets , you will have Ntrneq = Ntrn*O training equations to estimate Nw = (I+1)*H+(H+1)*O unknown weights. The estimation degrees of freedom is Ndof = Ntrneq  Nw (See Wikipedia).
The evaluation estimate MSEtrn = SSEtrn/Ntrneq is biased from using the same data to train and evaluate. To some extent the bias can be mitigated by using the degreeoffreedom"a"djusted value MSEtrna = SSEtrn/Ndof. Obviously, the result becomes untenable as Ndof > 0 because of the combination of too small Ntrn and/or too large H.
Real world data is usually contaminated by noise, measurement error, signal inteference numerical roundoff and truncation error. Therefore, if the number of equations equals the number of unknowns, the net will memorize a particular combination of signal and contamination and will not generalize well to other combinations.
A good approach is to have Ntrneq >> Nw and obtain a least squared error solution which tends to follow the signal and ignore the contamination. When Nw > Ntrneq, the net is said to be OVERFIT because there are more unknown weights than constraint equations to force a useful minimum error solution. However, if the signal dominates the contamination, an iterative solution will tend to pass through a useful solution on it's way to minimization via learning the contamination. Continuing beyond that point is called OVERTRAINING.
There are several ways to avoid overtraining an overfiit net. The most common are
% 1. Do not overfit % a. Increase Ntrn % and/or % b. Reduce H % 2. If overfit, do not overtrain % a. Use validation subset stopping % and/or % b. Use regularization (help trainbr, help msereg) .
Although a validation set is used in the design to stop overtraining, if it is large enough, it's performance will be a much better estimate of generalization error than that obtained from training data.
I use the R^2 statistic to evaluate nets. See Wikipedia and my various posts in the NEWSGROUP and ANSWERS. Search using Greg and one or two of my keywords R2a, R2trna, Nw, Neq, Ntrneq, Ndof
Also see the comp.ai.neuralnets FAQ and posts re overfitting/training.
>and through testing the network. But, in the neural network toolbox, there is validation . > also. What is the advantages of the validation inspite of training the samples?
Preventing OVERTRAINING. > Another remark is, I am using Matlab 2009b, instead i am also going to see if the validation criteria makes good result, but, i don't like to do randomization of the data because my training examples are almost time series. How could i used the data division in Matlab 2009b, since the net.dividefcn='dividetrain' does not support in my version. How can i use the data into train/validation/test without 'dividerand', which randomize data and not useful to me? . Only reason is that this randomization destroys my periodic series. Please, could you show sample code?
help/doc divideblock
help/doc divideind
ope this helps.
Greg



