Paul
Posts:
162
Registered:
12/7/09


Re: distribution of regression coefficients
Posted:
Nov 11, 2010 10:29 AM


On Nov 11, 5:45 am, "Rod" <rodrodrod...@hotmail.com> wrote: > "Ray Koopman" <koop...@sfu.ca> wrote in message > > news:cc3b56b7ce124f8c8a9ac5f2592f6e8b@n24g2000prj.googlegroups.com... > > > > > On Nov 10, 4:07 am, "Rod" <rodrodrod...@hotmail.com> wrote: > >> On Nov 10, 2:20 am, Ray Koopman <koop...@sfu.ca> wrote: > >>> On Nov 10, 12:44 am, "Rod" <rodrodrod...@hotmail.com> wrote: > > >>>> In regression y = a + b*x > > >>>> I know how to compute the covariance matrix for a and b. > >>>> I also know that a and b are normally distributed, > >>>> but what is the joint distribution of a and b? > >>>> Its tempting to guess bivariate normal > >>>> but I don't see how to show that. > > >>> y = X beta + e > > >>> W = (X'X)^1 X' > > >>> b = Wy > >>> = beta + We > > >>> If e is multivariate normal then so is We, and hence b. > > >> ditto a, but what of the joint distribution given that a and b are > >> correlated? > > > Sorry, I should have been more explicit (and used only lowascii > > characters). In what I wrote, X is a given n by p matrix > > of predictors, where n is the # of cases and p is the # of > > predictors, beta is a pvector of unknown coefficients, and e is > > a random nvector. If one of the columns of X is a dummy predictor > > whose value is 1 for every case then the corresponding element in > > beta is the intercept (your a ). So the intercept is "just another > > coefficient". > > > Whatever the distribution of e may be, if its mean vector and > > covariance matrix are m and S then the mean vector and covariance > > matrix of b are beta + Wm and WSW'. (Note: we usually assume > > m = [0,...,0]'.) > > > If e is multivariate normal then b is also multivariate normal. > > I rather hastily assumed my b was the same as yours, sorry. > OK I get it that if the e are normal then b is just a linear combination and > hence also normal. > Either I am not understanding what you are saying (likely), or you haven't > yet answered my question fully.
It's the former.
> To keep it simple lets keep the e normal and independent from each other. > Also let me return to my y=a+bx notation. > I am after the joint probability P(a,b) which because a and b are correlated > is different to the product of the two distributions for a and b separately. > I would put money on P being bivariate normal but for the life of me I can't > see how to work that out.
As Ray said, b = \beta + We, where W is computed from the X matrix. The theoretical variancecovariance matrix of b is E[(b\beta)(b \beta)'] = E[Wee'W']. Treat X, and therefore W, as constant with respect to the expectation. Since the e are assumed i.i.d. with zero mean and variance sigma^2, E[ee'] should be obvious (left to the reader as an exercise).
/Paul

