Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Topic: Error in Evaluation Criteria for Best Performance of the Neural Network
Replies: 1   Last Post: Mar 14, 2013 6:56 PM

 Greg Heath Posts: 6,373 Registered: 12/7/04
Re: Error in Evaluation Criteria for Best Performance of the Neural Network
Posted: Mar 14, 2013 6:56 PM

"Subodh Paudel" <subodhpaudel@gmail.com> wrote in message <khpfcb\$8p9\$1@newscl01ah.mathworks.com>...
> Dear All,
>
> I have find the optimum value of the hidden neurons, H=32, and then with Ntrials=5, i get the following result. How MATLAB neural network toolbox choose the best performance value.
>
> Nepochs R2Train R2Validation R2Test MSETrain MSEValidation MSETest
> 16 0.90386 0.84352 0.67618 0.096081 0.11011 0.11351
> 27 0.91307 0.8324 0.77425 0.086873 0.11793 0.079135
> 19 0.90546 0.81643 0.77273 0.094487 0.12916 0.079667
> 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566
> 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734

1. The MSE tabulations are not very useful for understanding since the scale of the data is unknown. One could combine the R2 and MSE tabulations to obtain the more revealing NMSE tabulations. But why bother when NMSE and R2 only differ by unity?

2. The R2trn tabulation values aren't satisfying since comparing them with R2val and R2tst indicate that OVERFIT nets have been OVERTRAINED in spite of using validation stopping.

3. Obtain the standard deviations of MSEval and MSEtst (You only have 5 samples so you might want to check with the CHI-SQUARE formula from Wikipedia instrad of repeating with a higher value for Ntrials). Then combine and compare with mean(MSEval)-mean(MSEtst) to see if it is reasonable to assume that the val and tst data are good representatives of random draws from the same population of data, much less the same population from which the trn data was drawn.

You may be the victim of a bad random trn/val/tst draw.

Did you have the same random draw of data for all of the runs?

4. It is hard to believe that H=32 is optimal. The minute I saw Hopt = 32, I felt that something was wrong. What are I, O, N, Ntrn, Nval and Ntst? The limit for overfitting is
the upper bound

Hub = -1 + ceil( (Ntrn*O-O)/(I + O + 1))

Overfitting is OK as long as you don't overtrain. A good validation set and/or using
MSE regularization should prevent that.

help mse
doc mse

5. What defaults did you override?

6. Forget the MSE tabulations but include the degree-of-freedom-adjusted training set R^2, R2trna. The comparisons of R2trna vs R2trn and R2trna vs R2val and R2tst are
VERY important.

7. To answer your original question, the best estimate of performance on unseen population data is R2tst at the maximum of R2val:

R2tst = 0.779 when R2val = 0.846

If this is unacceptable consider what I have said and ask more questions.

> With this result i need to check to check for MSE for the Validation to be minimum, and my best configuration should be, because i have MSEva 0.10809, which is minimum:
> 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566
>
> but when i check the performance from matlab with tr command,
> best_epoch: 25
> best_perf: 0.085477
> best_vperf: 0.10985
> best_tperf: 0.074734
> The matlab best performance is is best epoch 25.
> 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734
>
> How matlab choose this value? How to judge the best performance of the network.

That matlab best result is is only taking the last (5th) trial into consideration. The training structure results from previous trials is not available once you start a new trial.

> Also, the R-statistics from the matlab is
> Rtrain=0.95629, if we square it =0.91449
> Rvalidation=0.92598, if we square it =0.857438
> Rtest=0.91665, if we square it =0.84024
>
> These value are different.

Yes. I don't know why. They might be using different values for the MSE00s.

> How the best performance of the neural network is evaluated.