Thanks again for taking the time to write such a detailed response. I gave it a quice twice-over, but it needs more than that. I'll get back to it soon, but I got some preliminary comments.
On Nov 2, 2:02 am, Gottfried Helms <he...@uni-kassel.de> wrote: > it seems I made my comment more complicated than the procedure is. > > Am 01.11.2012 20:53 schrieb Paul: > >> I might be missing some linear algebra theory here, but I looked up >> gettrans() and I'm not sure what is meant by a column rotation in >> that context. > > No, gettrans is just a function-call in my MatMate-script-language, > which returns a rotation-matrix. For instance, by the command: > > t1 = gettrans(X,"drei") // "drei" means "triangular" > > t1 becomes the rotation-matrix, which is required to rotate > columnwise...
I am still not sure what is a columnwise rotation. Do you actually switch columns around, or is it more like a geometric rotation?
> ...X to triangular shape. After that we can do the following > with t1: > > Y = X * t1 > // Y is a lower triangular matrix, (with possibly empty columns > // to the right > Z = Y * t1' > // Z equals now X, because t1*t1' = I (Identitymatrix) > > or, for doing roation to principals components position: > > t2 = gettrans(X,"pc") // "pc" means "principal components" > > and then > > B = X * t2 > // the columns of B are now orthogonal, are the principal > // components > > I've introduced that function "gettrans" additionally to the simple > "rotate"- function to have the rotation-matrix available for later > manipulation, or to be able to reverse a rotation later or to apply > the same rotation to another matrix etc. It can also be made to work > only on certain columns and using only certain rows for the > criterion; This is then useful, if one uses rotations, which are > implemented as iterative procedures like "pc" or "varimax" or > similar. > >>> The key is, that the n samples define m vectors in an >>> n-dimensional euclidean space; simply each column of X can be seen >>> as a spatial dimension. In that n-dimensional space there are m >>> vectors, where the number m is smaller than n. Any rotation in >>> that space repositions the vectors, but *not* the relation, or >>> better: the angles, between them >> >> I'm not sure why *any* rotation in n-space would not preserve >> angles. I thought that a rotation is by definition a unitary >> transformation (from a recent brush-up on linear algebra at >> Wikipedia e.g. >>http://en.wikipedia.org/wiki/Orthogonal_matrix). > > My remark may be obfuscating here. There is the concept of "oblique > rotations" in factor analysis (as opposed to orthogonal rotations) > which do not preserve the angles - and I had the impulse to exclude > this case verbally... So this remark could just be deleted > >>> ...So we can rotate the vector model X >>> (columnwise) first such, that >>> sensor 1 defines the x-axes, >>> sensor 2 and 1 define the x-y-plane >>> sensor 3 to 1 define the x-y-z-space >>> and so on. > >> I don't quite follow what you mean by "rotat[ing] the vector >> [model] X columnwise". If you interpret each column of X as a >> point (or vector) in n-space, we get what you describe (sensor 1 is >> the x-axis, sensor 2 is the y-axis, etc.). However, a rotation is >> not needed for this. > > If we speak of the n-dimensional space, each column represent the > coordinates on one axis. Then each row represents one vector > (from the origin) to some point in this n-dimensional space: for > each sensor there is one wire from the origin into the n-space, > and the angles between that wires (more precisely: the cosines of > that angles) are expressed by the correlation-coefficients. That > view of statistical data may be somehow unusual - but it is coherent > with the operations of rotations and the finding of principal > components - and this is what your matrix Vt stands for. > >>> In effect, that rotation provides a matrix X1 which is triangular >>> with as many nonzero-columns as the rank of the matrix is (and we >>> assume for simplicityness, that it equals m) > >> I think I'm missing something fundamental...the data matrix is not >> triangular, though the (n)x(n) covariance matrix (Xt)(X) is >> symmetric. > > No, not the data matrix X. But after X is rotated to triangular > position by t1 then > X1 = X * t1 > is lower triangular (with some empty columns due to the defective > rank of X)
What is meant by rotating to triangular position? Do you mean geometric position, or that X somehow becomes a triangular matrix by rearranging its columns? What if there are not enough properly placed zeros for that to be possible?
>>> Then the matrix X1 can be rotated to the position of their >>> principal components (we're talking already of the nonzero columns >>> only), let's call this X2 >> >> I see that the data must be rotated so that the principal axes >> align with the axes of m-space (not n-space), and then the diagonal >> matrix Sigma performs the anisotropic axial stretching. > > No, again we rotate in the columns/the n-space. Just we apply the > (costly because of iterations) rotation to orthogonality (which > gives principal components) only to the first m axes in X1 (which is > already triangular with only m significant columns) > > X2 = X1 * t2 > or equivalently > X2 = X * t1 * t2 = X * (t1 * t2) = X * Vt > > After that X2 contains the coordinates of your sensor-measures > after rotation in the n-space in such a way that in the first > column the sum of squared coordinates is the maximum possible > and in the m'th column the least possible and because > X2 ' * X2 is diagonal we may say, that the columns are orthogonal > >>> That two rotations together form your matrix Vt. After that, X2 >>> can be rotated by rotation of its rows to diagonal form - this is >>> your rotation-matrix W, which rotates for the principal components >>> with respect of the rows in X2 (and which is the same as the >>> rotation with respect of the rows in X). >> >> But W is not applied after Vt, > > ??? > > If we have > W * X * Vt > we can also write > W * (X * Vt) > which is meant when I say that W is applied "after" the rotation by > Vt in my example....
I got lost...the middle matrix should be Sigma, a diagonal matrix of Eigenvalues.
>> So the rotation by W is very intuitive to me, while the rotation by >> Vt is not. And as I described, it's all the more mysterious when >> you consider that X isn't actually a transformation that is applied >> to data -- it *is* the data. > > This remark "... isn't actually a transformation..." confuses now > me. ;-) Well, I understood X as data as well, I have no idea, where > the idea of "being a transformation" comes from and what I am > possibly missing here. Very likely I didn't properly catch your way > of approaching the problem...
That's the view of X = W Sigma Vt. Sigma is an anisotropic axial stretch while W rotates these stretch axes to the principal components of the data in m-space. What is never explained is what Vt rotates. In order for the rotations and stretches to apply, X=W*Sigma*Vt must be viewed as a transformation applied to a vector (or a collection of column vectors). Which means Vt is first applied, then Sigma, then W. W and Vt are orthogonal rotations.
However, X isn't a transformation that is applied to data vectors, and it is hard to imagine what vectors Vt would apply to. They would have to be in n-space, but n-space doesn't have much meaning in the context of finding correlations between the m data sets (one from each sensor).
> -------------------------------------------------------- > (...) > >> Furthermore, when I am seeking correlation between the m sensors, >> it confounds me to think about why one would picture the data >> points in n- space. As an analogy, if I am doing simple linear >> regression on a cloud of 1000 points in the x-y plane, I don't try >> to picture the data points in 1000-dimension space. > > Well, we might say, such a concept is superfluous, not needed. It > just reflects a possibilitywhich occurs when we look at the > correlation matrix and its cholesky-factors. Say, with our m x n > -datamatrix X (I use the '-apostroph for transposition) > > R = X * X' / n // R is the m x m correlation-matrix > > then we have also with some rotation W > > Z = W * R * W' // Z = Sigma = diagonal > > but also, if we see R in its cholesky-factors L and L' > > Z = W * (L * L') * W' // Z = Sigma = diagonal > > and because any rotation-matrix t postmultiplied with its transpose > is the identity > > Z = W * (L * I * L') * W' = W * (L * t * t' * L') * W' > > Now L is usually taken as m x m matrix as well, but there is no > problem to expand it by empty columns to make a m x n matrix > out of it and then to assume t such that > > L * t = X / sqrt(n) > > and then rewrite: > > Z = W * (L * t * t' * L') * W' = W * (X * t' * t * X')/n * W' > > where again (X * t' * t * X')/n = X * X' /n = R shows the > identity of the solutions. > >>>  t1 = gettrans(X,"Drei") >>> t1 : >>> 0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937 >>> 0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963 >>> -0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206 >>> -0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151 >>> 0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125 >>> 0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685 > >> Sorry, I tried to google gettrans, but wasn't able to find much >> beyond the fact that it is a column rotation. It's not clear to me >> what is meant by that. Consequently, I wasn't able to follow the >> rest of the example. > > With the given parameters X and "Drei" (="triangular") it calls the > procedure, which returns that rotation-matrix, which can rotate X to > lower triangular shape. Having it stored as an explicite matrix we > can apply this rotation and also revert it and furtherly do anything > we want with it. > > If you are using windows, you can even download that MatMate-program > and do the steps yourself (and possibly experiment further) See my > software-pages http://go.helms-net.de/sw/matmate. It's an amateurish > program, however working nice for me, but if some installation > problems occur (which is easily possible) let me know.