Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
NCTM or The Math Forum.



Re: CROSSVALIDATION TRAINING FOR NEURAL NETWORKS
Posted:
Feb 15, 2013 4:27 PM


On Feb 15, 4:08 pm, "Greg Heath" <he...@alumni.brown.edu> wrote: > Although ffold XVAL is a very good and popular way for designing and > testing NNs, there is no MATLAB NNTBX function for doing so. I have > done it for classification and regression by brute force using do loops. > Unfortunately, that code is not available since my old computer crashed. > Nevertheless, the coding was straightforward because random weights > are automatically assigned when the obsolete functions NEWPR, NEWFIT > or NEWFF are created. With the current functions FITNET, PATTERNNET > and FEEDFORWARDNET, you either have to use a separate step with > CONFIGURE or let the TRAIN function initialize the weights. > > I did add a few modifications. With f >=3 partitions there are F = > f*(f1)/2 ( = 10 for f = 5) ways to choose a holdout nontraining pair for > validation and testing. So, for each of F (=10) nets I trained untill > BOTH holdout set errors were minimized. I then obtained 2 nontraining > estimates for error: The error of holdout2 at the minimum error of > holdout1 and vice versa. Therefore for f = 5, I get 2*F = 20 holdout > error estimates which is a reasonable value for obtaining > min/median/mean/std/max (or even histogram) summary statistics. > > However, I did get a query from one statistician who felt that somehow > the multiplication factor of F/f = (f1) ( = 4 for f=5) is biased. The > factor of f1 is from the use of f1 different validation sets for each > of f test sets. However, the f1 different validation sets correspond to > f1 different training sets, so I don't worry about it. > > In addition, for each of the F nets you can run Ntrials different weight > initializations. So, for f = 5, Ntrials = 5 you can get F*Ntrials = 100 > error estimates. > > With timeseries, preserving order and uniform spacing is essential. I > have only used ffold XVAL with f = 3 and either DIVIDEBLOCK or > DIVIDEINT types of data division. In the latter case the spacing is > tripled and success will depend on the difference of the significant > lags of the auto and/or cross correlation functions. The trick of two > holdout error estimates per net still works. Therefore, you can get > 2*Ntrials error estimates. I can't see any other way to preserve > order and uniform spacing. > > The MATLAB commands > > lookfor crossvalidation > lookfor 'cross validation' > lookfor validation > > may yield functions from other toolboxes which may be of use. > However I think the use of 2 holdout nontraining subsets is > unique to neural network training. > > Hope this helps. > > Greg



