The Math Forum

Ask Dr. Math - Questions and Answers from our Archives
Associated Topics || Dr. Math Home || Search Dr. Math

Line of Best Fit Using Least Squares Method

Date: 06/08/2005 at 00:12:07
From: Matthew
Subject: Least Squares for an Eighth Grader

In my geometry class, we were given a worksheet.  In each of the 
problems we were given several points and we had to find the best 
fitting line.

In a recent issue of "College Mathematics Journal" there were many 
references to the least squares method.  I've done some research and 
found that this can be used to find the best line.  Unfortunately, for 
the most part, the web sites I've used get a bit technical.  I
understand all the concepts (okay, I don't know how to work with
partial derivatives, but other than that...), but I haven't found the 
explanations very lucid.

Could you possibly explain the least squares method of linear
regression in simple terms that an eighth grader (who has taken
Algebra 2 and knows a bit about differentiation) could understand?

I researched the method on Wikipedia and Mathworld.  I also glanced 
through a calculus textbook that was lying around in my geometry 
classroom, but I couldn't find anything.

Date: 06/08/2005 at 10:46:27
From: Doctor Ian
Subject: Re: Least Squares for an Eighth Grader

Hi Matthew,

Here's one way to understand it.  Suppose you have a bunch of data
points, and you just draw _any_ line to represent that data.  (It
doesn't have to be a particularly good fit--in fact, it could be a
terrible fit.)

Now, from each point in the data, you can draw a vertical line segment
that connects the data point to the line.  The length of the segment
is the distance of the point from the line.  And you can add up all
those distances.  

If you do this with two different lines, the sum of the distances will
be different for the two lines.  In particular, for one of them it
will be smaller.  One way of comparing the goodness of fit for the
lines is to say that the smaller sum indicates the better fit. 

(Note that this is just one of many possible ways that we might make
such a comparison.  For example, instead of measuring distances
vertically, we might measure them perpendicular to the line.  But this
would be a lot more complicated, and it's not clear that it would
provide a 'better' measure.  But it would certainly provide a
different one.) 

One improvement we can make is that, instead of minimizing the sum of
the distances, we minimize the sum of the _squares_ of the distances--
it gives you the same result, but without having to compute a lot of
square roots, which are expensive operations.  

Does this make sense so far?  

Now we're in sort of the same situation we're in when we want to find
the square root of some number that isn't a perfect square, e.g., the
square root of 7429.  The simplest method is to make a guess, square
it, and let the result (is it too high, or too low?) tell us how to
make our next guess.  

Similarly, we could make guesses with lines (to make a guess, we have
to pick a slope and an intercept), and use those guesses to make
better guesses, until we get a fit that is "good enough".

Or, as we learn more math, we can start getting more sophisticated. 
In the case of square roots, we can use calculus to create techniques
that will move us more quickly towards the actual answer.  This works
by looking not just at how close our answer is, but at how much of a
change in the answer we'd expect to see if we change our guess by any
given amount.  (This is where derivatives--which measure rate of
change--enter the picture.)  So now we can move, not just in the
right direction, but by an optimal amount in that direction. 

Similarly in the case of least squares:  we can look at not just how
small the sum of squares is for the line that represents a given 
guess, but at how we would expect the sum to change if we change the
parameters that determine the line.

But that's the basic idea.  As a physical model, you can imagine that
each data point is represented by a pin stuck in a board, and your
line is a stick.  If you connect the stick to each pin with a rubber
band, there will be some way to place the stick so that the various
forces exerted by the rubber bands cancel out.  In any other
configuration, they'll be working to move the stick by pulling it or
twisting it.  

Because of the way rubber bands actually work, this would give you a
different answer than the least squares operation.  But in the sense
that rubber hands are "happier" when they're being stretched as little
as possible, it has the same sense of trying to minimize the lengths
of the rubber bands.  

Does this make sense?  Is it at the level you wanted? 

- Doctor Ian, The Math Forum 

Date: 06/08/2005 at 21:01:29
From: Matthew
Subject: Thank you (Least Squares for an Eighth Grader)

Yes, that was great!  Thank you very much!

Associated Topics:
High School Statistics

Search the Dr. Math Library:

Find items containing (put spaces between keywords):
Click only once for faster results:

[ Choose "whole words" when searching for a word like age.]

all keywords, in any order at least one, that exact phrase
parts of words whole words

Submit your own question to Dr. Math

[Privacy Policy] [Terms of Use]

Math Forum Home || Math Library || Quick Reference || Math Forum Search

Ask Dr. MathTM
© 1994- The Math Forum at NCTM. All rights reserved.