Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Threads in Neural Network in Train/Validation/Test
Replies: 1   Last Post: Feb 21, 2013 8:14 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View  
Greg Heath

Posts: 5,871
Registered: 12/7/04
Re: Threads in Neural Network in Train/Validation/Test
Posted: Feb 21, 2013 8:14 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Subodh Paudel" <subodhpaudel@gmail.com> wrote in message <kg5upc$pgj$1@newscl01ah.mathworks.com>...
> Dear All,
> I am using MATLAB R2009a. I have a different answer for train/validation/test from two different method:
>
> 1) I Use:
> [net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);
>
> to train the network, and simulate the different train/validation/test result as:
>
> normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
> normValidateOutput=sim(net,valV.P,[],[],valV.T);
> normTestOutput=sim(net,testV.P,[],[],testV.T);


Using norm in the output names is confusing because norm has a special meaning
(help/doc norm)
>
> and then i obtained MSE for training validation and test as:
>
> MSETrain=tr.perf(end);
> MSEValidate=tr.vperf(end);
> MSETest=tr.tperf(end);


I think if tr.stop indicates validation minimum stopping you should
replace end with end- tr.max_fail or tr.best_epoch.

> And from them finally R2 square value as:
>
> R2Train=1-NMSETrain
> R2Validate=1-NMSEValidate
> R2Test=1-NMSETest
>
> And the result i obtained directly from MSEtrain1=mse(normTrainOutput - tn(:XX)), that starts from training interval period i defined. And so on validation and test. Why these two values MSETrain1 and MSETrain differ?


If tr.stop indicates validation minimum stopping, then the last max_fail epochs should not be included. Find tr.best_epoch ,tr.best_perf, etc

> 2) I have R2 Train = 0.7738, R2 Validate = 0.7934 and R2 Test = 0.7926. And from the linear regression plot i obtain R train = 0.89584, R validate = 0.81805 and R Test = 0.92432. Does it mean the R2 value of neural network is worst than linear regression model? OR the result i obtained during training = 0.89584 from regression is quite good.

If you had chosen the val minimum epoch, I would have expected

R = sqrt( R^2 )
>
> 3) Every times i simulate my network, my R2 values sometimes good and sometimes even worst -ve. How to make it constant, if i assume i get 27 epochs, hidden neurons =18 the best R2 value?


You get different values because of the random data division and random weight initialization. If you intialize the random number generator to the same state (e.g.,
rng(4151941) ) before data division and weight initialization, you will reproduce runs.

I usually use a double loop over numH candidate values for H and Ntrials weight initialization runs to get five Ntrials X numH sized matrices for numepochs, R2trn,
R2trna, R2val and R2tst.

Search NEWSGROUP and ANSWERS for greg Ntrials (or other of my characteristic
variable names MSE00, Neq, Ntrneq, Nw, Hub, R2, R2a,...)

Hope this helps.

Greg



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.