Why Is the Least Squares Line the Best Choice?

Date: 11/03/2005 at 17:42:08
From: Jane
Subject: Least sqares: probability

When obtaining a prediction line for the relationship between certain 
variables, one method which may be used to identify the best line is 
to find the least-squares line.  The least squares line minimizes the 
error between the two variables.  What about the least squares line 
makes it best?

Thank you for your help, Dr. Math.

Date: 11/04/2005 at 14:41:14
From: Doctor George
Subject: Re: Least sqares: probability

Hi Jane,

Thanks for writing to Doctor Math.

If you plot your data it should be approximately straight, otherwise
there is no sense in looking for a best fit line.

There is nothing about least squares that makes it intrinsically THE
best fit.  A best fit line is only "best" based on some criterion, and
there are other possible criteria to use besides least squares.

The criterion is usually a matter of minimizing some quantity.  Least
squares minimizes a sum of squared distances.  Every other possible
line will have a greater sum of squared distances.  Other not uncommon
criteria are minimizing the sum of absolute distances, or minimizing
the maximum of the distances.

Does that make sense?  Write again if you need more help.

- Doctor George, The Math Forum
  http://mathforum.org/dr.math/