The Math Forum

Ask Dr. Math - Questions and Answers from our Archives
Associated Topics || Dr. Math Home || Search Dr. Math

Gradient and Maximum Increase of a Function

Date: 07/19/2005 at 04:58:42
From: Jill
Subject: Linear Algebra and Calculus

I am having difficulty explaining why the gradient points in the 
direction of the maximum increase of a function.  Several resources 
make the statement, but no one explains it.  It is probably something 
simple I am overlooking.  Can the dot product be used to justify it?


Date: 07/19/2005 at 05:54:56
From: Doctor Jerry
Subject: Re: Linear Algebra and Calculus

Hello Jill,

One explanantion uses the idea of "directional derivative," which
usually precedes the definition of the gradient.

If f is a function of two variables, if a = <a1,a2> is a point
(specified, for convenience, as a position vector) in the domain, and
if u = <u1,u2> is a unit vector (thought of as being based at a), then
the directional derivative of f at a and in the direction u is the
limit of the ratio 

  [ f(a+h*u) - f(a) ] / h

as h->0.  If this limit exists it is often denoted as D_u f(a), where
"_" means subscript.  This is the rate of change of f in the direction 
u.  It is easy to see that if u = <1,0> and <0,1>, the directional 
derivatives are the partials f_x(a) and f_y(a) of f at a, 

If f is differentiable at a, one can show that

  D_u f(a) = <f_x(a), f_y(a)> dot u. (dot = dot product of vectors)

Of course, <f_x(a), f_y(a)> is the gradient of f at a.  Recalling that
the dot product of two vectors is the product of their lengths and the
cosine of the angle between them, it follows that the maximum
directional derivative happens when u is the direction of the 
gradient, that is, the cosine of the angle between them is 0.

Please write back if my comments are not clear.

- Doctor Jerry, The Math Forum 

Date: 07/19/2005 at 11:48:09
From: Doctor George
Subject: Re: Linear Algebra and Calculus

Hi Jill,

Doctor Jerry gave a very nice answer to your question.  Here is
another approach to it.

Using Doctor Jerry's notation, if we examine the Taylor expansion of
the function, the linear term is

  h <f_x(a), f_y(a)> dot u

For sufficiently small values of "h" the linear term will dominate the
expansion, so the maximum increase in a small neighborhood about "a"
will occur when the linear term is maximized.  By Doctor Jerry's
reasoning, this happens when "u" is the direction of the gradient.

Just to broaden the picture a bit, in optimization theory the goal is
often the minimization of a function.  The direction opposite the
gradient is called the direction of "steepest descent."

- Doctor George, The Math Forum 

Date: 07/19/2005 at 13:43:29
From: Jill
Subject: Thank you (Linear Algebra and Calculus)

Thanks a bunch!  I appreciate you both taking the time to help!
Associated Topics:
College Calculus
College Linear Algebra

Search the Dr. Math Library:

Find items containing (put spaces between keywords):
Click only once for faster results:

[ Choose "whole words" when searching for a word like age.]

all keywords, in any order at least one, that exact phrase
parts of words whole words

Submit your own question to Dr. Math

[Privacy Policy] [Terms of Use]

Math Forum Home || Math Library || Quick Reference || Math Forum Search

Ask Dr. MathTM
© 1994- The Math Forum at NCTM. All rights reserved.