Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Non-linear optimization
Replies: 32   Last Post: Mar 8, 2013 2:22 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Matt J

Posts: 4,994
Registered: 11/28/09
Re: Non-linear optimization
Posted: Mar 8, 2013 12:18 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <khb5mn$7bg$1@newscl01ah.mathworks.com>...
> "Matt J" wrote in message <khb4uc$55k$1@newscl01ah.mathworks.com>...
>

> >
> > I don't see how that can be. For non-convex functions and non-posdef Hessians, an empirically chosen lambda could easily leave H+lambda*I singular, or at least, not positive definite and therefore non-descending. You would have to choose lambda>=min(eig(H)) to be sure that didn't happen, and that would require an eigen-analysis of H.

>
> The strategy and rules to chose lambda is given by LM algorithm. You can check the textbook that explains all the details.
>
> Again all that is well known.

===================

I've gone back to 2 textbooks now, Bertsekas and Nocedal&Wright. They both deal with LM strictly in the context of Gauss-Newton approximations for the Hessian in nonlinear least squares. I don't get the sense that LM for true Hessians has been explored all that extensively.

Neither book, incidentally, spells out the lambda-tuning procedure in great detail.
Nocedal and Wright in fact, offer a non-empirical alternative, derived from trust region ideas. Again, though, that's applicable only to Gauss-Newton LM.

One can see that the empirical lambda-tuning rules in the original LM papers should in theory be applicable to true Hessians and not require an eig(H) operation. However, it's still easy to imagine that if the algorithm lands in a non-convex region where H is not positive definite, that you might have to solve

(H+lambda*I)*x=-g

for several lambda before a descent-direction was found.



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.