Gradient and Maximum Increase of a Function

Date: 07/19/2005 at 04:58:42
From: Jill
Subject: Linear Algebra and Calculus

I am having difficulty explaining why the gradient points in the 
direction of the maximum increase of a function.  Several resources 
make the statement, but no one explains it.  It is probably something 
simple I am overlooking.  Can the dot product be used to justify it?

Thanks.

Date: 07/19/2005 at 05:54:56
From: Doctor Jerry
Subject: Re: Linear Algebra and Calculus

Hello Jill,

One explanantion uses the idea of "directional derivative," which
usually precedes the definition of the gradient.

If f is a function of two variables, if a = <a1,a2> is a point
(specified, for convenience, as a position vector) in the domain, and
if u = <u1,u2> is a unit vector (thought of as being based at a), then
the directional derivative of f at a and in the direction u is the
limit of the ratio 

  [ f(a+h*u) - f(a) ] / h

as h->0.  If this limit exists it is often denoted as D_u f(a), where
"_" means subscript.  This is the rate of change of f in the direction 
u.  It is easy to see that if u = <1,0> and <0,1>, the directional 
derivatives are the partials f_x(a) and f_y(a) of f at a, 
respectively.

If f is differentiable at a, one can show that

  D_u f(a) = <f_x(a), f_y(a)> dot u. (dot = dot product of vectors)

Of course, <f_x(a), f_y(a)> is the gradient of f at a.  Recalling that
the dot product of two vectors is the product of their lengths and the
cosine of the angle between them, it follows that the maximum
directional derivative happens when u is the direction of the 
gradient, that is, the cosine of the angle between them is 0.

Please write back if my comments are not clear.

- Doctor Jerry, The Math Forum
  http://mathforum.org/dr.math/ 

Date: 07/19/2005 at 11:48:09
From: Doctor George
Subject: Re: Linear Algebra and Calculus

Hi Jill,

Doctor Jerry gave a very nice answer to your question.  Here is
another approach to it.

Using Doctor Jerry's notation, if we examine the Taylor expansion of
the function, the linear term is

  h <f_x(a), f_y(a)> dot u

For sufficiently small values of "h" the linear term will dominate the
expansion, so the maximum increase in a small neighborhood about "a"
will occur when the linear term is maximized.  By Doctor Jerry's
reasoning, this happens when "u" is the direction of the gradient.

Just to broaden the picture a bit, in optimization theory the goal is
often the minimization of a function.  The direction opposite the
gradient is called the direction of "steepest descent."

- Doctor George, The Math Forum
  http://mathforum.org/dr.math/ 

Date: 07/19/2005 at 13:43:29
From: Jill
Subject: Thank you (Linear Algebra and Calculus)

Thanks a bunch!  I appreciate you both taking the time to help!