Eigenvectors of a Covariance Matrix

Date: 07/04/2004 at 21:34:22
From: Sesha
Subject: Eigenvectors of a Covariance Matrix

I understand the concepts of eigenvalues and eigenvectors.  My
question is why do the eigenvectors of a covariance matrix point in 
the direction of highest variance of the initial data set?

We create a covariance matrix using the data set.  Now, the covariance 
matrix becomes a transformation matrix, meaning if we multiply the 
eigenvector by the covariance matrix it would yield a multiple of the
eigenvector, and this multiplier is the eigenvalue.

Date: 07/06/2004 at 11:35:41
From: Doctor George
Subject: Re: Eigenvectors of a Covariance Matrix

Hi Sesha,

Thanks for writing to Doctor Math.

This is a great question.  The answer has to do with what is called a 
Rayleigh quotient.  You can get a theoretical explanation of this from
most any college linear algebra book.

Finding the directions of maximum and minimum variance turns out to be
the same as looking for the orthogonal least squares best fit line and 
plane of the data.  The sums of squares for that line and plane can be 
written in terms of the covariance matrix.  If you can work out the 
connections between them it should help you get an intuitive feel for 
your question.

Here are a couple articles on those subjects from our archives that 
may help you.

  Best-fitting Line to a Number of Points
    http://mathforum.org/library/drmath/view/51809.html 

  Orthogonal Distance Regression Planes
    http://mathforum.org/library/drmath/view/63765.html 

Write again if you need more help.

- Doctor George, The Math Forum
  http://mathforum.org/dr.math/